There's no such thing as a "Safari Filevault Password" - maybe you mean "keychain password." Open the Keychain utility, and you should be able to see all of your passwords. It will ask you for your master keychain password (usually the same as your account password) to see it. You can also just delete all the Safari entries in your keychain too.
But it sounds like your download folder is set to a folder where you don't have permission to write, so it's asking for a password. Check in the Safari preferences to see where your download folder is.
Lars is right, but thanks for giving me an opportunity to repeat my favourite warning: don't use Filevault unless you are involved with our nation's national security.
Lose that password (or, as often happens, the correct password gets corrupted and won't work) and you are SOL.
You've been warned.
+1 on Filevault. It makes absolutely no sense to encrypt and decrypt everything when only a very few files need to be secure. Use DiskUtility to create an encrypted disk image for that handful of files.
Personally I also tend to avoid Keychain. In this case I store passwords in a file which is kept on an encrypted disk image. That is the only password I really have to remember but I find it best to be able to remember all my passwords as a corrupt Keychain file can result in all key-chained passwords being lost.
Yes Keychain passwords can and do suffer from corruption.
I retain all rights to images I have posted on ehMac. They were posted that other members of the community could enjoy them. They may not be used or sold in any other way without my written consent.