The bank has confirmed it is to permanently close its branch on the High Street this summer.
A spokesman stated the increase in online and telephone banking as among the reasons for the closure from 19 August.
He said: 'We will be working with customers who use the affected branches to help them understand their options, which include being able to use local post offices for their day-to-day banking, and using local ATMs.'
The town's Natwest Bank branch closed down last year and this latest announcement has raised concerns about the impact this could have on the town centre.
Simon Prior, the chairman of Downham Market's 'Town Team' said: 'This is very disappointing news. It is the second bank within a year that has closed in the town.
'Banks are an important feature of our high streets and also bring people to shop our town - the closure of HSBC in Downham Market is not a good thing.'
Elsewhere in Norfolk, HSBC closed branches in Wroxham and Attleborough earlier this year.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here