Could an AI truly be conscious?
Or would it just behave in such a way, while not actually 'feeling' anything?
I can't see how we could ever prove or disprove it.
No. AI is programmed to do a specific task. The programmers make them do what they want. Hence artificial in AI. Think you've been watching too much sci fi.
I think you are missing the point.
What is the difference between a real brain, and an artificial brain constructed atom by atom to mimic the real thing?
Would this artificial brain not 'develop' consciousness once it reached a certain level of complexity?
If not, then what are you suggesting is missing?
A 'soul'?
I don't think so, and the main reason is that our bodies are the source or feedback loop for a lot of our "feelings" and we're not going to the trouble of creating AI with a body like our own. There are other reasons I don't think it will happen, but you're right about one thing, we wouldn't be able to prove or disprove it. Read Antonio Damasio if you want some in-depth ideas on the brain and feelings.
The singularity.
We'll see this within a decade.