Overlapping colliders and OnMouseUpAsButton

Hi I am pretty new to Unity and I'm not sure if this is the right forum, but anyway.

I am making a simple tower defense game. The game grid is made of buildplaces (cubes) which all have a collider in order for towers to be built on them.

This was all fine until I started implementing tower upgrades. The towers now need to be clicked on to be selected which means they no longer ignore raycast. But since their colliders have large radii (for detecting enemies) it means I can no longer click on surrounding buildplaces.

Is there any easy way to fix this? I tried creating different layers but that didnt seem to work and as a beginner I am out of ideas.

 

Use thwo different colliders. one for selecting towers, and one for detecting enemies. Sorry about the phone fingers, this forum just doesn't work on my phone.

 

Thank you ThermalFusion, I didnt think of that!