R
Link preview
LLMs know when they are wrong. I made a fix relating to Anthropic's new "global workspace" paper [R]
I have posted before about finding out a model's actual confidence in its answer through probes and hidden states (AUROC ~0.83–0.88 across every model I tested, 7B to 72B). This is the know-say gap. From my work and the work done by others in this space it is likely a routing problem. By making a tiny bridge from a linear probe on mid-layer sate plus ten trained weights that write the probe's estimate onto the confidence-digit logits can make the model verbalise calibrated confidencve at 0.765+. No weights modified, answer never changes, needs about 200 labelled examples. It also doesn't matter when you install it: before alignment, after, or bolted onto a finished model. The gap is a routing problem, not a capability problem. Anthopics paper (https://www.anthropic.com/research/global-workspace) relates to this. They show models have a small "verbalizable workspace" (the J-space). It is a privileged subspace holding the concepts the model can report and reason with, sitting on top of a much larger ocean of processing that it can't report. This is possibly the know-say gap's anatomy, preventing it from reaching speech. My controller is basically way to route around it. I am planning to dig a bit deeper into this but I wanted to share the paper as I through it was relevant (its been on hold with ARXIV for over a week but here is the zenodo link - https://zenodo.org/records/21237443 Code and pre-registration links are in the paper. submitted by /u/Synthium- [link] [Kommentare] reddit.com · reddit.com ↗
I have posted before about finding out a model's actual confidence in its answer through probes and hidden states (AUROC ~0.83–0.88 across every model I tested, 7B to 72B). This is the know-say gap. From my work and the work done by others in this space it is likely a routing problem. By making a tiny bridge from a linear probe on mid-layer sate plus ten trained weights that write the probe's estimate onto the confidence-digit logits can make the model verbalise calibrated confidencve at 0.765+. No weights modified, answer never changes, needs about 200 labelled examples. It also doesn't matter when you install it: before alignment, after, or bolted onto a finished model. The gap is a routing problem, not a capability problem. Anthopics paper (https://www.anthropic.com/research/global-workspace) relates to this. They show models have a small "verbalizable workspace" (the J-space). It is a privileged subspace holding the concepts the model can report and reason with, sitting on top of a much larger ocean of processing that it can't report. This is possibly the know-say gap's anatomy, preventing it from reaching speech. My controller is basically way to route around it. I am planning to dig a bit deeper into this but I wanted to share the paper as I through it was relevant (its been on hold with ARXIV for over a week but here is the zenodo link - https://zenodo.org/records/21237443 Code and pre-registration links are in the paper. submitted by /u/Synthium- [link] [Kommentare]
Comments