I had a brainstorm today about a possible prompt to test the edges of LLM conpetence. It's longer prompt than most of the ones I use and demands a longer answer, so context lengths have to be expanded and it runs slowly on available hardware. As a result I've only begun investigating it.
But it triggered an idea for another "off the wall humor" type question which is quite succinct. The thing is that when I started asking models about that one I got a surprise, and thinking it might be perculiar to a particular model family I tried some others; and the surprise persisted. What. The. Absolute. Heck?
There is an old saw that a boat is a hole in the water that you throw money into, which is not the core of my prompt but has a similar flavor. What I'm actually asking the models to react to is a pair of definitions of two classes of boats in terms of how they make you broke. If you've spent anytime in boat-owning circles you've probably heard it. The joke is funny because (a) it acknowleges that boats are conspicuously expecnsive to own, and (b) it distinguishes between motor boats and sail bots, and you can order the definitions so your audience feels good about not being in one category until you hit them with the definition for the class of boat they own. Then it's rueful smiles all 'round.
Only ... model after model has wanted to interpret it as some kind of financial metephor. The difference between spending big for lots of velocity and binding your time doing things right in the hopes of achieving a better outcome. Or something like that.
Where are they getting that? Does their training data inclue lots of articles from the popular business press? Or something?
Weird.
Oddly it seems to come outstronger when models are allowed to reason.
No comments:
Post a Comment