Land-use planning

A setback rule is measured from the legal parcel boundary. A building appears to be set back from a nearby fence, but the fence does not mark that boundary. What should the reviewer rely on?