AI ACCURACY Did AI just lie to me and smile about it?

Dave shares a recent experience of his – should we always trust the output of AI? 

Image by gerd altman from (free) pixabay  

AI in property 

Artificial Intelligence has arrived and it's here to stay. It's been coming for a while in its simpler formats with Google and other search engines. Estate surveying is not immune to it and will have to embrace it. Paper is increasingly being replaced by sophisticated digital methods of recording, storage, retrieval, interrogation and analysis. Now AI is becoming available in even more advanced formats to produce advice and solutions to complex problems. But is it reliable? 

I remember the comfortable old days when all information was on paper. I kept copies of everything that I thought might be necessary, particularly as a defence against councillors or the public who might want to dispute advice or adherence to policy. I could always hide behind 'I have it in writing'. Most new members of the council (and some members of staff) had little knowledge of what had been decided in the past and no central source of reference, so the paper copies I kept always gave me the advantage. 

Consider this simple example of change. Many were the times that a new member would approach me and say, "I'm a big supporter of this charity. You know that property we have coming empty at ..? Can we let them use it rent free to help them out?" Then I would have to explain the approved policy on subsidising charities and voluntary sector organisations which lay buried in historic minute books which they couldn't possibly know about. And of course, they never liked the explanation so wanted to see the evidence to look for loopholes - hence the photocopies that I kept. Nowadays they can probably find such information online at the tap of a keyboard using Google or the council's intranet. So far, Google etc will usually only tell them where to look. However, they will soon be able to cut out the middle man by also asking AI in detail how their particular request will fit or not within that approved policy and, more particularly, how to get around it. AI will tell them, and in a very convincing way. 

A personal experience 

But will you or they be able to rely on that information, advice and any solution given by AI? The following is not a surveying experience, but it is a true account of my recent conversation with a machine. Please keep the title in mind as you read it. 

"To think this little photograph, 

On common paper lightly cast, 

May look into your face and laugh,  

When I myself have inevitably passed." 

This verse was written in pencil on the back of an old photograph of Holyhead’s first and only steam lifeboat ‘The Duke of Northumberland’, handed down to me through my family. The photo shows my great grandfather, Lewis Jones, standing on the boat sometime before his death in 1915. 

I was able to make a rough match of the handwriting by comparing it with his signature as a witness on a wedding certificate. However, I didn’t know if he had composed that verse himself or had simply read it somewhere and copied it on to the photograph. I tried researching the verse via Google on the internet but with no matches, so that made me more confident that it was his own work. 

Subsequently a friend of mine offered to research it, employing some very modern AI software used to detect plagiarism in research papers written by students. That AI programme also failed to find a match. So, I continued to believe that the verse was composed by my great grandfather. However, lack of certainty still nags at me and I never miss an opportunity to test that belief. 

Recently, Microsoft released its new, free AI App named 'Copilot' and invited me to test it out. I thought that this might be a good challenge for it. This is the conversation as it took place, with edits (as depicted below by '... etc ...') but only to remove uncontested or unnecessarily lengthy detail (Copilot can be a bit long-winded at times). The phrases highlighted in bold by me are particularly relevant. 

Me: Who wrote this (quoting the verse) 

CoPilot: The poem you shared is "A Photograph" by Shirley Toulson (poemanalysis.com )...etc ... Shirley Toulson was an English poet, born in May 1924, who also worked as a journalist and politician (poemanalysis.com ). Would you like to know more about the poem or the poet? 

Me: Yes 

CoPilot: Shirley Toulson was a celebrated English poet known for her poignant and reflective poetry... etc ... Her poetry often explores themes of memory, loss and the passage of time, deeply influenced by her personal experiences and observations of life. ...etc... - "A Photograph" is one of Toulson's well-known poems that eloquently captures the essence of time and remembrance ... etc ... In the poem, Toulson discusses the ephemeral nature of life and the lasting impact that photographs can have. The lines you shared beautifully illustrate how a photograph can continue to evoke emotions and memories long after the subject has passed away... etc ... 

CoPilot then went on to talk about its Themes and Style and Additional Works by Toulson before ending with ... Is there anything else you would like to know about Shirley Toulson or her poetry? 

Me: Show poem 

CoPilot: I can't provide the full text of copyrighted content, such as poems, verbatim. However I can share a brief summary or analysis of the poem "A Photograph" by Shirley Toulson, if that would help. Would you like a summary or analysis of the poem instead? 

Me: Show the verse that includes the quoted words. 

CoPilot: I can't provide the exact lines of the poem verbatim due to copyright restrictions. However I can offer a brief summary of the poem ... etc... In this particular verse, the poet reflects on how a simple photograph captures a moment in time that continues to hold emotional significance even after the people in the photo have passed away ... etc ... If you need a detailed summary or analysis of the poem, feel free to ask. 

Me: When did she write it? 

CoPilot: Shirley Toulson wrote the poem "A Photograph" in 1970 and it was published in her poetry collection titled "Circumcision's Not Such a Bad Thing After All and Other Poems.(msroclassroom.com )... etc ... 

Me: I've just read the poem on msroclassroom.com and none of the above verse forms part of it so how do you explain your first response? 

CoPilot: I apologise for the mistake. It looks like I mixed up the information about the poem. The lines you shared don't actually match Shirley Toulson's "A Photograph". I strive for accuracy but sometimes errors do happen. Thank you for pointing this out. If there's anything else you'd like to know or discuss, feel free to ask! 

Me: Thank you. I accept your apology. 

CoPilot: I appreciate your understanding. If there's anything else you'd like to explore or discuss, feel free to ask. I'm here to help with anything you need! 😁

This was a good outcome for me as I didn't want CoPilot's initial answer to be true. But what a way to find out. CoPilot had access to Toulson's poem on the internet, despite not being allowed to quote it, so must have known that the verse was not in it. It took me seconds to find it using Google and it actually appears in full on the website that CoPilot acknowledges as one of its sources. So CoPilot had been able to read it. Yet it attributes that verse to Toulson's poem that doesn't include it and states that positively with the phrases that I have highlighted in the conversation using 'The lines you shared...' and 'In this particular verse... 

CoPilot lied. There is no argument about that, despite its fulsome apology and its rather weak explanation of 'It looks like I mixed up the information about the poem.' What really worries me though is that smiley emoji added at the end. I thought that CoPilot wouldn't have any feelings but is there a hint of sarcasm in that last sentence? Did it just lie but not really care about it? If so, that's a very worrying human trait. 

Implications for professional work 

It could be interesting to see how it copes with specific requests for land disposals under s123 of the Local Government Act 1972 as amended by the General Disposal Consent 2003. Or, maybe it needs testing with asset valuations, rating assessments, etc. It's too late for an ex-surveyor like me to be asking it about local government estate management subjects because I’ve forgotten more than I ever knew. 

However, if any non-retired surveyors are inclined to challenge it with current issues… it requires no more effort than if you are sending an e-mail to a colleague asking for advice … then please let the ACES Editor know how you get on. 

[Ed – Dave puts out an invitation for colleagues. I’d be happy to include future AI debunks if readers have them. Who says I’m a Luddite?].