I'm going to go ahead and rule the following: 

The asylum dates back to 1915, it served for quite a number of years as a sanitarium and psychiatric ward. Due to hurricanes repeatedly bashing it, the place became run down. A lot of patients were removed either by their family or case workers. In an attempt to revitalize the business (it is a business after all) they applied for renovation funding with the state. They were awarded a good sum of funding and construction began. Many of the patients were removed, but some of the more docile and clearer ones were allowed to stay if they chose. The place ran on minimal staffing, nothing more than a skeleton crew. It was a relief for some, and they did have the ability to get way from the banging and sawing and noise. 

Perhaps Brandt even worked at this construction site, helping to restore and improve the building. The new construction seems to be building on the old which had been built quite sturdy to begin with. Construction crew abandoned the project as soon as the outbreak started, seeking to get home to their families. A couple staff members may still be there, looking after the small population left over, no more than five or six... However.. they might have been overrun and turned into walkers. The staff could have fled, and could have taken the few remaining residents with them to somewhere else.

I do intend for there to be a grounds keeper/landscaper still on the property, holed up in a small workshop, sectioned off. The entire compound is surrounded by a chain link fence, topped with barbed wire angled in and out on the top. There's also an iron square tube bar fence running around the property set into a low one foot wall of brick and concrete,, with two-foot high crenelations every ten feet. It's decorative in look, but it's hard to climb, even for the living. The top features arrow or spear type tips an inch or so wide and about two inches long. Not exactly sharp on the edges but the points would not be fun to slip onto. This fence, again, is hard to climb, is much sturdier than the cyclone fence, and could possibly hook and slow down any geek that did manage to get to the top. Fence is 10', both of them. There's an iron gate at the front with a buzzer, but that's not active now and would need to be secured with a chain and lock or carabiner or something.