The 72-year-old Welsh crooner had been due to entertain tens of thousands of people at the outdoor concert in Hyde Park on Saturday. He was replaced by British singer Will Young.
Jones tweeted “sorry to all,” adding that doctors had ordered him to cancel gigs scheduled for Saturday and Sunday.
He later added: “Thanks for all the lovely messages, I don’t know if everyone’s understanding and support is making me feel better or worse! You’re the best!”
The concert was part of the BT London Live series that is offering fans without Olympic tickets an opportunity to listen to music and watch sports on big screens.