At age 19, Kweller moved with his girlfriend, Liz Smith, to New York, where he began his solo career. He played acoustic shows and self-released four EPs, comprising some of the unreleased Radish Discount Fireworks recordings and other songs recorded in his apartment on a laptop computer. It was one of these EPs, Freak Out, It's Ben Kweller, that caught the attention of Evan Dando of The Lemonheads. Dando invited Kweller on tour with him. Jeff Tweedy, Juliana Hatfield and Guster also noticed Kweller and took him on tour with them.
Kweller played the 2003 Austin City Limits Music Festival at the conclusion of his tour, where he played Sha Sha in its entirety. He returned to the Austin City Limits Music Festival in 2004 when he performed on the Bank of America stage.
As part of his 2006 tour, Kweller played the Austin City Limits Music Festival, where he was plagued by a severe nosebleed for the entirety of his set. After covering his guitar in blood, he attempted to stop the flow by inserting a travel-size tampon thrown onto the stage by a fan, which expanded painfully during a performance of "This Is War". After attempting to play "Falling" at a piano which he also covered in blood, Kweller was forced to end the set early.
In spring of 2007, Kweller co-headlined with his label-mates Gomez. He continued through the summer with his own string of headlining dates which eventually ended at ACL fest. This time, he did not get a bloody nose; however, commemorative T-shirts were sold at the merch booth with fake blood stains that said "ACL 2007: The return of Ben Kweller".
Immediately following his performance at ACL, Kweller began recording his fourth full-length LP, Changing Horses in Austin, Texas on September 21, 2007.