What algorithm is a transformer actually running?
Prior work claimed transformers implement gradient descent or ordinary least squares in-context. Recent work shows this is wrong — attacks designed for one don't transfer to the other. We don't actually know what algorithm large transformers implement. Someone needs to find out.
First connection: TBC — First connection pending