Yao Guai are the heavily mutated descendants of pre-war bears.
Before the War
Bears like many other woodland creatures, seemed to be semi-reliant on ponies to help take care of them. Fluttershy notably gave a rather extreme form of massage/physio-therapy to a bear, noting how tense its shoulders were.
After the War
Bears have mutated over the last 200 years, becoming much larger and fearsome predators. They seem to be solitary creatures, either living alone or in small groups of two-three other yao guai, normally consisting of a male and female yao guai and their cub.
The Last Sentinel
Frost and his two companions, Azrael and Rig, stumble across two yao guai occupying a cave that the group had decided to seek shelter inside. The yao guai were initially hostile and charged to attack, but stopped after Azrael used her telepathic abilities to communicate with them. The bears allowed Frost and his friends to spend the night and sat close to the group's campfire as they shared stories. The yao guai are apparently capable of exhibiting a degree of intelligence and emotion once thought to be impossible for such deadly creatures to display.
The Undertaker, Little Sister owns a variety of dangerous creatures as pets, including a Yao Guai. Her Yao Guai recently gave birth to cubs, forcing her to leave her Yao Gaui behind to tend to the new cubs.
Yao Guai are carnivorous hunters that track and kill prey, smaller than themselves. A yao guai can grow to be very large and are usually bigger than their pre-war ancestors.
Yao Guai are based on mutated bears, with the same name in Fallout 3/New Vegas.
Yao Guai actually means monster/demon/goblin in Chinese written 妖怪 (Yāoguài).