It's because everyone has noticed the changes in Lingyi since she entered the family. They all see the influence she has over him. He even started standing up to his mother because of her so these are significant things/that's the reason why they continue to blame her. If you remember when his late wife was alive she controlled the family/he was worn out/got tired of arguing with her. So during that time he didn't really put much value on the household matters. His mother even controlled his movements/decided on his marriages. Lingyi lost his fighting spirit dealing with all those venomous women including his mother. They didn't value his opinion/they didn't care what he wanted. Shiyi was different. She took out the time to listen to him/cared about what he wanted. She changed the way he interacted with his family. She helped him build a relationship with his sons. She helped him reconcile with his brother. It was easier for them to blame her because she was the one responsible for the changes in him. It drove me crazy every time they blamed her. It was as if they didn't believe that Lingyi had a mind of his own. Concubine Wen should've been grateful to Shiyi because she was the reason that Lingyi took an interest in Yu. She was also the reason he wanted to send Yu to Leshan to study. He didn't want her family to use their son. He even asked his son before he prepared to send him. So it was Yu's choice to go/if Shiyi hadn''t told him to talk to Yu first things might've worked out differently. She was still upset that he walked out on her birthday...lol