Its like all things in IT - AI is the current shiny new trend. Just like IoT (Internet of Things), Big Data and the Dotcom bubble were a few years ago.
Once the shiny new trend, and all of the associated hype, has passed it will settle down to a more sustainable level and hopefully become a positive, properly regulated, contributor to society.
Either that or it will use up all of the planet’s resources or unwittingly start World War 3.
Yes, this is a huge problem that isn’t really being properly addressed. As AI models develop, over a very short time, they become more and more resource hungry.
I often wonder whether the Jolla Mind 2, limited in non-cloud mode to what you could just get locally and securely in the box, was obsolete before it was even produced.
No it doesn’t. You stated:
I responded by saying that the output of algorithmic computer programs (of which an AI model structure is a complex version of) can neither be random, nor unpredictable, nor imprecise because of the nature of a computer simply being a machine that follows a set of instructions programmed into it by a human. That is not conjecture, nor opinion, but established fact. I further suggested to you that if the output of a computer program, be it an AI model or any other program, was incorrect or unexpected then this would be the fault of the human programming it, not the fault of the computer (hardware failures aside) or the software it was running.
I only used the example of a computer ‘random number generator’ as … an example of this, nothing more.
I should’ve added a 3rd option: I don’t really care.
Is the app useful/dangerous to [me]?
I always find it amusing when people want to be againt LLM but come up with basically a modern version of vitalism. What makes you think human mind have superior functionality? Delete your post and re-write it again, do you think it will be the same?
If the ultimate input/output end, i.e. human, is random and imprecise, I don’t see why an additional layer of random and imprecise will fundamentally make anything worse (as far as my past experience, SOTA LLMs are already way better than quite some of the programmers I met). They are at least much predictable than the-guy-who-work-remote-from-a-different-country-and-pushed-the-code-with-force. It ends up the same question, whether you can have test facility/audit process good enough to check the code generated by LLM as we used to check the code generated by human.
It is OK to use AI, as long as the developer takes (some) responseability for the final app. Also, learning how to actually code is a great way to use AI as well. And of course, provide the source and explain what vision/idea they have for the app in question.
It is not the randomness itself that clearly makes LLMs inferior compared to a trained programmer. LLMs are functionally just sophisticated autocomplete models. Harnesses can improve them by making sure aspects where they are especially bad are taken over by non-LLM functions.
The key problem is that based on how LLMs work they are incapable of actual reasoning and abstract understanding. They are incredibly good at predicting how text has to look like to be in line with basically all previous knowledge on the internet, though. It turns out that is enough to write somewhat functional code, especially for common tasks. All models struggle to some extend with following instructions however and that is due to fundamental issues of what LLMs are.
Ask specialists on machine learning and they can probably tell you better why LLMs might be superior in some tasks but are overall much more limited in their width of capabilities than the human mind. That has nothing to do with vitalism. You would not think that pocket calculators are superior to the human mind just because they can do one thing really well, do you?
All that said, LLMs can be helpful tools for coding, but without human oversight and review issues will appear at some point. for anything moderately complex.
I think you are missing @attah’s point. He is addressing the comparison of LLMs to other development tools from the history of computing. Previous tools were deterministic. Compile the same file twice and get the same output. This enables important safety measures like reproducible builds.
Skilled developers can produce quality code faster with careful use of AI tools. Unskilled or careless people can produce lots of plausible code very fast, but with subtle or disastrous problems embedded and difficult to audit. Hey, it even generated test cases for me, and THEY ALL PASS!!!
I wouldn’t seek medical advice from a layperson with an LLM, and I’m just as hesitant to use an LLM coded app unless I have some proof that the developer knows what the code does.
This is just plain wrong. Matrix math on GPUs produces randomness. The temperature settings for an LLM, even when set to 0 (ie. deterministic output) STILL display randomness when using GPUs. And as the ‘thinking models’ are much less interesting/ useful in their output with a 0 temp, they are generally LESS deterministic in general use.
As to training an expert model, I have some experience (automotive) building on existing models with a custom data set (Safety regulations). Now, we didn’t build from scratch but used an existing (visual encoder/decoder) model with our data. Three years later it’s clear that chat gpt with relatively hight temperature (.3) is more capable than what we built with a carefully delimited (and large) dataset. The less deterministic system is more useful. The prompting is probably the most carefully engineered on the planet (the engineer in question also being a capable programmer). This does not eliminate errors. Neither did a custom model (never mind the performance element which is hard to keep up with).
I stepped out of the project, but it’s run by a friend who has fined tuned the new system running with chatgpt. His estimate is that it cuts the document generation phase from weeks to days. Every document needs to be examined by two qualified (certified) engineers in any case. And errors are found. That’s not important when you’ve saved weeks in the production phase.
So, there is imprecise and sometimes unpredictable output from LLMs. Even with the best prompting by qualified experts in the field WITH the engineering know how.
EDIT: since I get worked up with claims about infallibility … explain to the layman WHY floating point operations are not exact at limits… I learned how to deal with the ERRORS in floating point using Fixed Point math (which is rarely done except in an embedded context these days). What I know, both of myself and many others, is that the error prone end of floating point math is ‘bodged’ around (most games have numerous ‘workarounds’). Why, because it is easier to just use the fpu than to handcraft the fixed point math. So, long before LLMs there have been sources of indeterminate / undefined behaviour that litter the codbases of many, many projects.
OK, I’ll try - others can jump in if I get anything wrong.
The vast majority of computers today store numbers, instructions, text, etc in binary powers of 2, in other words base 2 representation. So the first bit of a binary byte represents 1, the second bit 2, third bit 4, fourth bit 8, and so on. Depending upon how many bytes you use to store a whole (integer) number dictates the largest number you can store.
So, in a single byte the biggest number you can store is 1 + 2 + 4 + 8 + 16 + 32 + 64 + 128 = 255. If you had two bytes you could store a bigger number (1 + 2 + 4 + 8 + 16 + 32 + 64 + 128 + 256 + 512 + 1024 + 2048 + 4096 + 8192 + 16384 + 32768 = 65535), three bytes an even larger number and so on. So, no inaccuracies and job done. Either the hardware designer, or the software programmer can decide how many bytes are used to store the number.
Binary is very convenient for electronic computers, because each bit only has two states - 0 or 1 - beautifully represented by no voltage or a positive voltage in a binary bit circuit.
For a moment lets assume our computer is a base 10 computer, not a base 2 computer - so each bit stores a power of 10 (1, 10, 100, 1000, 10000, 100000 and so on). Now each bit can have 10 states - the first bit 0 to 9, the second bit 10 to 99 and so on. Again, for whole numbers, not a problem - depending upon how many bits you use you can store any whole number.
But here’s the rub - whatever number of bits is chosen to store the whole number, in whatever base, it obviously has to be a fixed number of bits (or bytes). It cannot be an infinite number. The problem with floating point operations is when you are dealing with fractions.
In our base 10 computer, lets consider a simple sum - 1 divided by 3. We all know the answer, its one third. If we multiply one third by 3 we get back to 1 right?
But our base 10 computer can’t do that because it has to express and store everything as a decimal number. When it tries to divide 1 by 3 it gets 0.333333333333333333 (and so on forever - its called ‘recurring’). And that’s a real problem because we have to store that decimal number in a fixed number of bits, and you cannot store a recurring decimal fraction in a fixed number of bits.
So what to do?
Well there are only two solutions - you either truncate to x decimal places (e.g. 0.33333 to five decimal places, or you round up or down (e.g. 0.33334)). But whatever you do, the answer is going to be inaccurate (either wildly inaccurate if you truncate to, say 0.3, or slightly inaccurate if you truncate to 0.3333333333333333333333333333333333333). The more decimal places you work to (or number of bits you have to store the decimal number), the more accurate your answer to the sum is going to be.
Now take your truncated decimal and multitply it by 3. You should get back to 1 right? But you won’t, you’ll only get back to 0.99999. The more you keep on dividing and multiplying, the more inaccurate the answer will become. So, if you are calculating a missile trajectory and target that might not be a good thing!
But here’s why @poetaster’s ask is a good example. Although the results of doing floating point math on a computer is inaccurate and results in errors, it is neither random, nor unpredictable. If you ask a Base 10 computer to divide 1 by 3 and truncate to 5 decimal places - you’re always going to get 0.33333, and not 0.32333, or 0.33334, or whatever. Because of these inaccuracies in floating point math, and as @poetaster says, programmers have been coding in bodges and workarounds for years to cope. If these inaccuracies were random and unpredictable then you wouldn’t be able to do these things with any degree of success.
But why? the concept of a ‘temperature’ variable in an LLM is a human designed concept to moderate (or not) the ‘creativity’ of the LLM’s output. The point here is that the LLM still does not operate in a random way (but may look like it does) and the output is ‘human unpredictable’ only in so far that the algorithms of the inference engine are so many and so hugely complicated that it becomes a mammoth (and probably pointless) task to follow all of the reasoning pathways and predict what the output might be.
Dating right back to the 1970’s, and largely driven by the US Government Department of Defence (perhaps calculating those missile trajectories again?), there have been efforts to develop systems to mathematically prove that computer programs do what they are expected to do, had no bugs, and delivered predictable outputs based on specific inputs.
This was touted as an alternative approach to the more traditional human error-prone test & validation of human developed software.
To a certain extent this strategy also got caught up in the Philip Crosby cost vs quality initiatives of the early 1980s - the theory here being that the earlier you caught a bug in the software lifecycle, the cheaper it is to fix (i.e. during alpha or regression testing very cheap, during beta testing/trials more expensive, during released and live software use hugely expensive). Mathematically proving software was error free would pretty much guarantee production software quality in a way that human testing could never do.
When I last was involved with this as part of my Masters degree in the early 1990’s ‘mathematical software proving’, even after 20 years of trying, was still in its infancy. But there now seems to be some renewed interest in it and its usage with AI models in helping ensure their outputs are predictable in a way that they don’t suffer from hallucinations and create plainly fantastical outputs. I have no idea how much progress they’ve made though.
You’ve made my point for me. A layman zoned out before the end of the first paragraph.
Your explanation is complete but for one point. We have the number of bits these days to use Fix Point math. So the ‘what to do’ question can plainly be answered. Don’t naively use floating point. The degree of success could be much improved upon if only…
Bugs related to ‘work arounds’ in the treatment of floating point numbers have killed people. If the reliable inaccuracy of floating point math is not dealt with responsibly, will this not be doubly and thrice so of the use of LLMs.
I’ll state it again, since you misunderstood. Regardless of ‘temperature’ setting, you will get ‘random’ calculations caused by deficiencies in GPUs (and similar vector/matrix specialized processors). They are not as easy to account for as fpu architecture limitations. In the case of LLMs, which are stochastic in any case, this can generally be ignored, but cannot be discounted and the effects of which are still only poorly understood.
A stochastic mask/screen is also a useful way of creating the illusion of continuous color tones in color lithography. But it is an illusion. Look under a magnifying glass, and you will see. But it is a very good illusion and may look remarkably like a ‘real’ photographic print (which is of course an other kind of illusion). The problem with LLMs is that the work of discerning where they ‘go off the rails’ requires more effort than just looking through the glass to spot the dots.
Even so, the algorithms are not so many and not so huge. It all fits in Francois Fleuret’s Little Book of Deep Learning. Gradient Descent, propagation (back propagation), activation functions, etc. None of these things are beyond a computer science undergrad. I have no issues understanding that machinery. I can even use verbose output to see the stream of ‘reasoning’ in thinking mode which is generally easy enough to understand even when it’s misguided. What is huge is the corpus.
Which brings me to English Common Law. Among legal systems, that system is a continuum over a millennia in the making. Jurisprudence in England is of greater complexity as a consequence than that in the United States or Germany. Why? Because the corpus is much larger and the effort to interpret it faced with greater obstacles (the evolution of the language, etc, centuries of precedents). I very much doubt the integrity of a legal bot made in the UK, without the commensurate and radical act of throwing away a thousand years of codification and interpretation. There may come a time, but it is not yet.
This is true and some of the progress has to do with the mathematical community (with LEAN and coque before it) meeting some of the more mathematically inclined languages (haskell, for instance) with strong typing, etc. I think you are correct that this will eventually lead to better LLMs and other kinds of reasoning systems. Mathematicians like Terrence Tao working toward a systematic LLM use with LEAN as the proof system are certainly promising.
But the state of the art LLM corps are not actually aiming for this. The aim is something like ‘duck hunting’. With shotguns. I always found that distasteful.
Well, you asked me to explain and I did my best. I’m pretty sure that I would have no chance of explaining why errors occur in floating point math done on a computer to a layman (who knows nothing about computers or how they work) in one short paragraph before that layman would zone out. If you can do that then I tip my hat to you and will definately buy you a beer next time I am in Germany.
No, I don’t believe I misunderstood you or the position you were advancing - I just don’t agree with it. My position is, simply stated, that a computer is a dumb machine that can only follow the instructions it is programmed with. It is therefore wholly deterministic and is incapable of producing truly random outputs (hardware failures aside). I understand that AI models like ChatGPT work on probabilities and statistics, but that is because a human programmed them that way. I also understand that you will get …
and I presume ‘random’ is in quotes because it is not really, well, random but pseudo-random. But again a human has decided to integrate that functionality into the computer’s design and that such errors in calculations are acceptable - or at least the benefits of doing so outweigh the downsides.
So, in a nutshell, my position is that computers, AI models, etc are only capable of producing pseudo-random or non-deterministic results because a human has designed them that way. They cannot ‘choose’ to do these things on their own because that would imply real intelligence rather than artificial intelligence. And the introduction of, how shall we put it, ‘AI uncertainty’ only just adds to the uncertainty of the human interacting with it.
So, as gentlemen, we are going to have to agree to disagree.
Well that may be exaggerating a bit. Some common law here in the UK does date back that far, but most is only hundreds of years old. One of my own areas of legal specialisation is contract law, and that only really and formally dates back to the law of Assumpsit established in the late 1300’s/1400’s. Similarly the Statute of Frauds was established in 1677, and the definition of murder only dates back to 1797 (if I remember correctly), the exact same definition being still in use today.
What is perhaps more important for AI and common law cases is that court reports, the official trial records, have only really reliably been recorded since the late 17th Century. Before that time many trials went unrecorded completely, or if they were recorded the voracity was somewhat suspect and dependent on the judge’s, ahem, vested interests and bias.
So if you are going to train an AI model on UK court reports you would only sensibly go back 300 years or so. And of course a lot of the early precedents established several hundred years ago have long been superceded by more modern ones, so are no longer relevant.
And yes, you are right, legal language has changed over that period (e.g. an ‘assumpsit’ is now a ‘contract’, a ‘plaintiff’ is now a ‘claimant’, the ‘prisoner at the bar’ is now the ‘defendant’, etc) but not so much that an AI model couldn’t be trained on those differences.
So I think, yes, an AI model to act as a reliable ‘electronic litigant’, or claimant or whatever is still a long way off, but from the work I am doing there is a place for an AI model in legal research. Often here, because there are so many cases, statutes, secondary legisllation, precedents, etc, you don’t really know where to start looking.
Thirty years ago you would have trawled through endless books, and probably missed loads of what you needed. Ten years ago it would have been a simple database search. With AI you can ask it to search and identify ‘intelligently’ and within a context, say for all cases relating to whatever. Now I wouldn’t use the AI’s output directly (many lawyers have and found themselves in very hot water), but as a first pass search to identify likely cases that I would then read up on myself it could save 100’s of hours of time.
And the programmer-human decided to make it pseudo-random. The operator-human has little say in it.
Please stop it with the semantics and take the argument at face value - LLMs output variations unrelated to improving things.
This is a fundamental departure from everything that came before it.
Of course not.
I will endeavor to improve.
But just because humans work that way it doesn’t automagically mean that it is a desirable trait of tooling - rather the opposite in fact. I want my tooling to be precise and predictable, in part because i am not.
Variation without improvement is a bug, not a feature.
At least you can eventually lock this guy out. AI seems to be entirely excused.
The problem here is that we generally don’t have a test and audit process that is good enough to check all humans - and we don’t have humans good enough to check the AI after the fact. Review is a really poor replacement for brain-on work from people who knows what they are doing.
This. Exactly this.
This is not an argument for or against AI (LLMs really) as such, but only that if we treat it just like any other tool we’re setting ourselves up for a very different set of problems than ever before. Many of which we are poorly equipped to handle.
If you ask the same human to code the same thing twice, they will write different code each time. So will AI. But that doesn’t matter!
Which was my point. The GPU errors I was referring to are in large part determined by hardware issues.
Anyone who has ever had to take the not so deterministic ADC of any of a number of systems to make it more so will recognize that a good deal of physical force may be required (a diode clamp will often do). Not only are the ‘error conditions’ of the programming a good source for random, but the hardware itself. I’ll go dig up the papers in question.
EDIT: there is a certain irony in this discussion for me, on this day. I have, at some point last year, written code which is ONLY deterministic when certain compiler optimizations are employed ( gcc, -O2) and the cpu frequency ist set just so (276mHz). So, the intended behaviour is a side effect. Now, not random, certainly, just really strange code and a strange co-incidence in the side effects produced by the compiler.
Indeed. The Elliott 803 had magnetic core memory arranged in 40 bit x 40 bit planes. There were X and Y wires attached to each plane and threaded through each magnetic core (obviously one core representing one bit of the 39 bit + parity bit word). These were used to magnetise the cores in either direction to represent 1’s or 0s. There was also a sense wire running diagonally through the plane. At the end of each X and Y wire there was a simple discrete component voltage amplifier (no microchips then!). The pertinent (to this story) components of each amplifier were a germanium transistor and a 91 Ohm bias resistor.
In the Summer of 1976 (you can see where this is going) the 803 started lighting up like a Christmas tree with parity errors all over the place. Program errors ensued and nothing worked properly. This was in the era when computers were ‘real computers’ and had lots of flashing lights, would fill a room, would draw 3KW of electricity to run, and so on.
Old germanium transistors, unlike modern silicon ones, are very sensitive to permanent temperature induced drift - and the very hot summer had caused the bias voltage to drift so far that a lot of magnetic core 0’s were incorrectly being read as 1’s. I well remember spending days un-soldering dozens and dozens of 91 Ohm resistors and re-soldering in 89 Ohm resistors to restore the correct bias voltage in each affected amplifer.
So you see I am quite familiar with a temperature variable introducing hardware failure induced random behaviour, only in this case it was the real temperature, not a made up coined parameter.
Ah, well, then you will well understand that some of us are STILL occupied with such duties in an age where no one will believe you when you point out that the hardware itself is liable to physical disturbances which are ‘glossed over’ by layer upon layer of indirection in software, which does not always cut it.
It is quite often that I throw fruit at electrical engineers who do everything in simulation.
Still sounds vitalism to me. Define “actual reasoning”, this is typical No true Scotsman fallancy.
Yes I am aware, and that’s the reason I have a second paragraph explaining we should treat the problem as AI similiar to problem of terrible human developers. On second read, I can see I failed to explain myself, sorry for this. Maybe following can clarify a bit:
I have a tendency to starting to believe that deterministic abstraction has it limits and we may have well being close to that limit (or maybe Lisp have already reached there in last century).
I have to admit when implying the similarity here, I also tend to think AI can (and should) be used to check upon AI as well. I tend to think AI can enhance experienced human auditor at least as much as AI can enhance inexperienced coder.
curious, why do you believe that? there is no inherent reason for precise, deterministic models not to operate perfectly at any desired level of abstraction. building those models is hard, and running an LLM is easy; is that all you meant?
why would running anytying through an LLM twice be more likely to make it good? wouldn’t it be better to improve the parameters on the first pass?
–
my main practical problem with vibe coding is the potential for it to make the endusers into inadvertent script kiddies. there is a large incentive to get vibe coders to include attacks unknowingly, and a large possibility of accidental or malicious privacy violation that even low-skilled human developers wouldn’t do by accident.
then there’s licensing; vibecoders and ‘real’ developers (even using AI as a guided, thoughtful tool) can violate software licenses unwittingly. pulling GPL’d code into non-GPL or even proprietary projects happens all the time. (this is similar to the ethical debate of ‘AI art’)
reliable source traceability is impossible in current LLMs, and will likely require a technological revolution on the order of the attention matrix, or a complete restructuring of how we build the corpora of the next generation of AI tech.
and of course, vibecoded software is a nightmare to debug and maintain, but i agree that humans can be worse when they (don’t) put their mind to it. the ease of vibecoding just means we will have more of it.