NotNorm,
One of the ways older homes were quite often built is referred to 'balloon framing'. This technique often created continuous wall cavities from basement to attic. If there is any way for you to check in the closet you describe to see if this is the framing style used and if so if fireblocking was installed. If this is missing you will have each one of your wall cavities create a chimney effect.

Modern building is 'Platform' where each floor sits on top of the walls below and the floor extends all the way out to the outer sheathing and seals the stud cavities. This framing still allows for the condition you describe where air gets in the wall cavity on the first floor and moves up the the underside the the second floor and then across under the floor.

You will have to start at the bottom and work your way up and around your house to seal all voids and stop the drafts. It is not an easy or quick process but it is required to make your home comfortable year-round.
Good luck, Jim
'Just a handyman trying to help'