ChatGPT for SW Dev: Strengths and Weaknesses

ChatGPT is great for code development ... if you need some small, focused functionality for which inputs and outputs can be clearly specified. It is also good a helping you to analyze and develop system requirements at a high level. But where it really sucks is at software maintenance of an even moderately large codebase. The problem lies in that, without it being able to grok a large number of tokens, it is not going to really understand your codebase. And worse than that, it thinks it does really understand it. In that regard it is like a junior engineer who thinks he knows more than he does, and is simply unable or unwilling to admit when he doesn't understand something.

A case in point: I tried to get ChatGPT's help in modifying some Django python code for my spaghettispeller.com website. Most of the changes were in frontend files, specifically javascript. Changes needed to be made in several different modules and it appeared to work on localhost dev machine. So I pushed it to Github and then to the webserver. Well, there were a couple of bugs that didn't show up during localhost testing, so I went back and worked with ChatGPT to debug.

To make a long story short, I went through many revisions to no avail, and ended up chewing the AI out for not remembering changes it had made earlier in the day. After several frustrating hours and updates and test, I finally threw in the towel and used git to roll back to a previously working baseline (thank heavens for git!!!!) and will now start again. I should have known better: I've been burned many times by ChatGPT's overconfidence in its products. But, you get impatient and ... bingo you've got a bug that's hard to find.