Former high Court Judge Michael Kirby, who is also a prominent campaigner for the lesbian, gay, bisexual, transgender and intersex (LGBTI) community, said some residents faced hostility and hatred earlier in life and feared the reaction of fellow residents.
"The horrible thing is that some people who go into aged care have to revisit the whole process of coming out," he said.
"They thought they'd been there and done that, but it still is a journey they've got to make again.
"Somehow we've got to make sure that doesn't happen."
One organisation trying to tackle the issue is Australia's largest faith-based aged care provider Uniting.
It has become the first large operator to receive the Rainbow Tick accreditation, which recognises safe and inclusive service delivery for LGBTI people, for its LGBTI policies.
"If you want to disclose your sexual orientation or gender identity you can, and you can do it safely. If you don't want to, that's fine," said Steve Teulan, Uniting's director of ageing.
The Rainbow Tick was developed by Gay and Lesbian Health Victoria and the health charity Quality Innovation Performance.
All of Uniting's 6,500 employees underwent training and assessors surveyed 30 Uniting sites as part of the Rainbow Tick process.
"How do you create privacy and respect and dignity for people? And create a community that is truly inclusive?" Mr Teulan said.
"There's a lot work that goes into that."
Providing aged care and ageing services for the LGBTI community has long been a challenge.
But having LGBTI policies in place is a good start, according to Dianne Latter, a gay nurse who works in a nursing home.
"The main thing is to be aware that some LGBTI people haven't disclosed yet," she said.
"The generation we're caring for currently, they have felt discrimination.
"They're naturally hesitant to disclose, and there will be issues around that."
One resident the ABC spoke to said they had been openly gay for decades before entering aged care.
Now, they are reluctant to reveal their sexuality because of the fear of how fellow residents might react.
Justice Michael Kirby said that was a legitimate concern.
"It's also true that some of the other residents grew up with all kinds of prejudices and attitudes and hostility, and even hatred towards gay people," Mr Kirby said.
"There's a lot to be done, and it's not easy."
Some of those living with the assistance of Uniting said the LGBTI policies were making things better. Warren Talbot is a gay man who moved into an Independent Living Unit run by Uniting in 2014.
"Knowing that there was a policy, and even the application forms I filled in, there was a reference to LGBTI people," Mr Talbot said.
"I felt quite relieved and comfortable that that aspect of my life wasn't going to be an issue."
Michael Kirby believes the LGBTI issues in aged care will fade with time.
"It tends to be a last lingering generation that has the attitudes of prejudice," he said.
"You've just got to get over your demons. That is, I think, the challenge in aged care. Help older people who have prejudice get over their demons."