Ally Pally and photos.

Hi,

I was wondering if someone could tell me am I right or wrong in this.

I went up to ally pally recently to take some photos, photography is a hobby of mine,  on this occasion I took a tripod, while I was taking a photo of the outside of the pub side of the building a security guard asked me did I have a permit to take photos, I said no, why would I need one, he said its private property and you are using a professional camera. I said it's not a professional camera, nor am I a professional, the photos are for me, it's a hobby, he asked me to leave, I wasn't bothered getting too confrontational so just left, although I did ask him a few questions...I said where can I not take photos of and he said the building, I said so the view is OK to shoot, he said yes, I asked is it OK to take photos of the building with a phone camera, he said yes, so I asked why not with the camera, he said you can't with the camera, it's on a 'stand'. I left. As I said I couldn't be bothered getting too confrontational as I wasn't in the mood and if somebody accuses me of breaking the law I'd just ask them to call the police but this seemed like an over reaction so left it at that. 

So anyways, does anyone know, should I have left, can I go back to take photos, I thought Ally Pally was owned by the council, it's not someone's house! Any suggestions most appreciated. Thanks.