How to make my TOTP generator work with other hashing algorithms (other than sha1)?

I'm having trouble making my python code generate the right TOTPs using the test secret key "12345678901234567890" the same one used in the RFC6238 appendix-B examples, I only got the correct codes with "sha1" but was not able to do so with other algorithms "sha256" and "sha512".
I think what I'm doing wrong is the secret key encoding, I use it raw :

secret = "12345678901234567890".encode()

I used this before:

secret = base64.b32encode(b"12345678901234567890")
but it gives wrong codes as well.
I'm not very used to this (security related coding), so I don't really understand the format the secret key needs to be in, so I think the problem is mostly related to how I initially coded the secret key before using it in the hmac function.

I followed the algorithm so I doubt there are problems in the generating process since it also gave the correct codes for "sha1".

I would really appreciate it if you guys could help me out with this !