Solution: In order for video tag to work in Safari on Windows, it is required to have the Apple QuickTime installed too. Period.
Interesting thing about it is that Safari’s console will tell nothing about it. Video will just never show up.
It would be also interesting to see how Chrome and Firefox are dealing with video codec support/management.