I think it determines on who gets the first shot, I'm more on Lucci on this one. After seeing the fight between Luffy and Lucci, Neji's feats doesn't stand up to Lucci's in terms of physical power.
EDIT: I don't think Neji stands a chance, Lucci survived Luffy's gear 3rd punch which destroyed the building they were in, and the effects of the punch literally didn't look like it phased him all that much.
Considering the fact that Lucci is the only character in the entire OP series to bring Luffy which is nearly immune to physical force because of his rubber body to the brink of death through pure physical force? Neji is out of his league on this one.