I'm still not fully sure why so many have so much issues with framerate, sure I drop a little as well, but it's barely noticeable.
But I somewhat understand it, I mean you are drawing doublets of everything, the game already draws a player for you, so when you are i.e. using a bounding box or the like, then you are doubling the workload for your computer.
But as I said, it seems to be drivers and certain hardware that causes it, because my computer is about 7 years old now and I still max most games out there.
Not saying it's great any more, but seeing specs of some of you guys (it's better than mine) and you still suffer.
Which makes me believe it's more of a driver issue or background processes.