Annoying Rattle

I'm 99% sure that the Front Passenger Door has either been replaced or the locking mechanism has been replaced before I owned the car and since moving house there is an annoying rattle on most road surfaces. (road surfaces are worse around here than Wakefield). The door doesn't shut easily and does require a bit of a shove to close compared to the other doors which is making me think that something isnt lined up properly but the outer lines all match up.

Initialy I thought it was the door panel as 3 of the clips were broken and the panel moved around, these were replaced and the rattle continued.

Then when sound deadening the doors I had a good feel around inside the door panel but couldn't feel anything loose or laying in the bottom of the door skin, I also checked the door latch and that was bolted tightly, before putting the door back together I placed bluetack on the electrical connectors to stop them moving about, this did seem to cure the rattle but it has returned and now worse than before

My next avenue of investigation is leading me to remove the door panel and drive around without one fitted to see if its the door latch cable or something else.

Am I correct in thinking that removing the door panel and closing the door will only allow the door to be opened from the outside?

Does anyone have any other thoughts on what to check for whats making the rattle?

Matthew