Thanks Neo for the suggestions.
In the end I had to pick the drop down next to delete and select "Ignore Conversation" to be able to get rid of it.
Note, I only access O365 via the browser (Firefox).
It is resolved now, but this might be useful for anyone else experiencing this issue.
Cheers, James