I agree. Stated another way, imagine the trolley is headed towards 5 people, and you have the power to pull the lever to divert it to a path where there are no people. Even if someone tied those 5 to the tracks with the intent to kill, your failure to save their lives (at no additional cost to others) is widely regarded by most systems of ethics/morality as a moral failing. Yes, the person who tied the tracks bears blame, but so does the person who could've easily saved them but chose to let them die.