Strange terrain performance: snapping the edges to the lowest point is causing slowdown

I've got a very large, 10000x10000x1500 terrain, using a 2048 heightmap generated by World Machine. In the game you fly around, so flying off the edge is an issue. One way I thought of improving this slightly is to have the very edges of the map all snap down to the lowest point, so at least if you go off the edge you don't get the strange "under the level" effect which is very confusing for players not expecting it.

To do this I loaded the raw file into photoshop, and put a 1px wide square of black pixels around the edge of the map. Loaded it into unity, and it worked fine. It looks pretty strange with the textures stretched like mad, but better than seeing through the level.

After compiling, the framerate is now significantly lower, even when I'm nowhere near the edge of the terrain. Camera.Render takes about 7.5ms as opposed to 2.5ms before.

Why would this happen? Where can I get information on how terrain is rendered?

Thanks.