The number of restraints per residue is not as important as the distribution of restraints. The overall quality of a structure, in a case like this, depends critically on the distribution of NOe's among the following catergories: long range, medium range and short range (in terms of residues). If you have a sufficient number of long range NOe's, then a reasonably good model can be built from the long range NOe's only, using restrained energy minimization (or restrained molecular dynamics/SA) with a good force field. Remember that, in such a case a complete force field such as AMBER, CHARMM, etc has to be used, rather than the minimal force field that we use normally for NMR structure determination.
You could try to remove ALL intra-residue and sequential NOe's from your data and see if there are any violations when you generate a structure. If there are no violations using only long range data then you MAY be able to get away with a recalibration. However, if you observe violations, even with the long range restraints data set, then you have no option other than the use of ensemble based optimization methods.
Violations may be due to errors in assignment, due to calibration or due to multiple conformations sampled by the peptide. If you have multiple global conformations, then ensemble optimization is unavoidable. However, if a single major conformation exists, but local motion is significant then the problem is with your calibration. This is because, in the presence of motion, each class of NOe's has to be calibrated separately - in some dynamical models the distance dependence of the NOe's is no longer inversely proportional to r^6. Models ranging from 1/(r^3) ... to 1/(r^6) have been proposed for different types of models.
To assess the extent of conformational flexibility, it would be highly advisable to obtain additional data regarding J-coupling constants, H/D exchange rates, relaxation rates, etc.