I prune my friends list regularly, though unless there's something I'm missing, it's a lot harder to prune for people who don't sit at your table than it is to prune for people who don't aid. Maybe about half my friends will sit at my table during a day on an average day, and I always clear them shortly after it fills up. I've had maxed out tables in 6 different cities, I've pruned my friends list in all of them, and I usually get about the same number of people sitting in all of them, so I don't think it's all that unusual of a ratio.
I do indeed normally pull in about 2 full tables, a partial, and what I get from sitting at other people's tables. So, no, 3500 isn't impossible, but it is roughly my average day, I think. (Honestly, I'm not 100% sure how much I pull in in a day because once you have a maxed out table, you really don't need to worry about tavern silver ever again.) My point, however, was that this isn't something that I have full control over, even with pruning, and there's also a lot of people who can't log in several times a day to clear their table like I can.
Yes, challenges are supposed to be challenging, but to me, a challenge, especially something called a "Daily Challenge" is something where you have to do something a little special that day. There is absolutely nothing I can do in a given day to alter how many people sit at my table (except spam my friends with messages they probably don't want and would unfriend me for sending).
Also, most notably, this recently jumped from 2500 to 3500 for me. If that's the result of adding a few friends, it's a pretty big jump. If it's based on an average of what I normally pull in and can do as long as I'm paying attention, then I withdraw my complaint.