The key here is the frequency, not the number of visitors.  A small number of people visiting the building weekly would meet this definition. However, a large number visiting once a year (e.g. for an open day) would not.

However, it is important to note that visitors only have to visit part of the building.  They do not need to have access to all of it to be considered a visitor.  Someone visiting a reception area, however small, would be considered to be visiting the building.

Employees, residents of the building, suppliers, contractors or specially invited people (e.g. job applicants invited for interview) are not considered to be members of the public. However, students at an institution are considered to be members of the public.  Equally, residents in a complex visiting other buildings on the site may be considered to be members of the public.  In a healthcare setting, patients are considered to be members of the public.