Four small fires were lit near food stalls at Queen Victoria Market about 11pm.

CCTV footage appears to show a man using a blow torch to set fire to a tarp covering a fruit juice stall on the Peel Street end of the market in the CBD.

The man then threatened a member of the public who tried to put out the fires before emergency services arrived, police allege.

None of the stalls were open at the time, but tens of thousands of dollars worth of damage was caused.

A 38-year-old man of no fixed addressed was arrested at the scene and has been charged with criminal damage by fire and unlawful assault.

He was remanded to appear at Melbourne Magistrates Court today.

Police are working to notify market stall owners of the damage.

Running since the 1870s, Queen Victoria Market is known for fresh fruit and veg stalls and is the largest open air market in the southern hemisphere.