Not rather popular amongst
Mistrusted I was
After the murder of the Riddles, Frank was arrested and questioned in connection with the murders. As there was lack of evidence, the police could not prove that the Riddles had been murdered, and were forced to release Bryce. However, the community of Little Hangleton was distrustful of Bryce. As a result, he lived out the rest of his life by himself, living on the grounds of the Riddles' estate, while still working in the grounds.
More than once I was seen
The second a mere shadow
Frank appears first at the beginning of the Goblet of Fire, overhearing Voldemort's plan to kill Harry. But before he can take action, he is killed by Voldemort. He appears once more in the same book, during Harry and Voldemort's graveyard duel. The Priori Incantatem effect brings back an "echo" of Bryce from the tip of Voldemort's wand. Despite not knowing who he is, Bryce enthusiastically encourages Harry to keep fighting.
Lived near but not a part of anything remotely magical
Although he worked as the caretaker for the Riddle whose son is the father of Voldemort, he never had anything to do with the magical world until his death.
Now have a guess as to who I am
Frank Bryce