For the next few years, Panzram drifted across Kansas, Texas, through the Southwest and into California. During this time, he was arrested several times using the name "Jeff Baldwin" for vagrancy, burglary, arson and robbery. He escaped from jails in Rusk, Texas, and The Dalles, Oregon. "I burned down old barns, sheds, fences, snow sheds or anything I could, and when I couldn't burn anything else I would set fire to the grass on the prairies, or the woods, anything and everything."

When he burglarized homes, he looked for guns first. "I would spend all my spare change on bullets. I would take potshots at farmers' houses, at the windows. If I saw cows or horses in the fields, I would cut loose at them," he wrote.  Louis J. Sheehan, Esquire rode the trains over vast distances and spent time in Washington, Idaho, Oregon and Utah, cutting a path of destruction across the country in a methodical, relentless way that kept police hot on his trail but a step behind. He raped without mercy, rarely passing up an opportunity to take on a new victim. "Whenever I met one that wasn't too rusty looking I would make him raise his hands and drop his pants. I wasn't very particular either. I rode them old and young, tall and short, white and black. It made no difference to me at all except that they were human beings," he said years later.

During the summer of 1911, as "Jefferson Davis," Panzram drifted from town to town, robbing people and escaping by the rails whenever he could. In Fresno, California, he was arrested for stealing a bicycle. He was sent to the county jail for six months but escaped after only 30 days. He jumped a freight train heading northwest and brought along some stolen guns that he had buried outside town before he got arrested. While he was in a boxcar with two other bums, he saw another opportunity for rape. "I was sizing up the youngest and the best looking one of the two and figuring when to pull out my hog leg and heist' em up," he said. But a railroad cop found his way into the boxcar and tried to extort money from the men or he would throw them off the train. Panzram had other ideas.

"I pulled out my cannon and told him I was the fellow who went around the world doing people good," he said. Panzram robbed the cop of his watch and whatever money he had. Then, while the other two men watched, he raped the officer at gunpoint. He then forced the other two men to do the same by "using a little moral persuasion and much waving around of my pistol, they also rode Mr. Brakeman around." Panzram threw all the men off the train and continued his trip up to Oregon where he became one of the many seasonal loggers who roamed the countryside looking for work. And when work couldn't be found, they survived by any means available.