Am I overthinking this or is implementing secure email OTP auth basically impossible?

or your just offloading your entire auth to the persons personal, phishable, email account, the security of which you have no control over. Perfectly valid response if you don't care about the additional risk from user breaches & put it on your end user's for the CUECs.

Why do you want to do that? Personally I really hate that kind of login, I always have to wait for the Mail and often it gets stuck in spam. Why don’t you want to implement something like passkey or some kind of sso with whatever service?

then you are indeed overcomplicating it, accept the risk that an attacker could lock an account out and be done. Locking an account out isnt a flaw, the security risk in this situation is a compromised email is all an attacker needs, theres no way for you to verify what an “email owner” is, if they have access to the email, they are the owner from your POV.

"Email, if properly configured sender and receiver" - something that applies to about 0.01% of all email users. You are right on the potential drawbacks being similar though. At any rate, OP clarified he wasn't talking about using it for OTP.

Sometimes cybersecurity is a tradeoff between usability/design and security. In your post you already noticed that your top via mail is not really secure and that is right. Email alone is not a really secure channel. In my opinion your solution is not even a trade off because I think email opt has nothing to do with a great ux.

Are you now overcomplicating things? User signs in with username password and gets a code by mail or SMS. For true Email OTP just drop the password. After 3-5 wrong attempts, lock for X minutes. Or am I missing something?

SMS would have the exact same problems that OP described. Email, if properly configured sender and receiver, isn't less secure the SMS. Google Authenticator, while more secure, is generally less user friendly for adoption.

You have no guarantee the user email isn't already compromised.

You said now that your site is a low stakes website and not a "crypto wallet". Then we have the answer, https://www.reddit.com/r/cybersecurity/comments/1vbq4fr/comment/p0vchle/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

For this wouldn’t you just use captcha or something similar?

Are you only use OTP authenticating (essentially magic link) or is the default 2FA after a password? For larger service providers, they can take advantage of network effects, eg they pay for ddos protection and have their use paid or their own blocklist that protects a large number of customers (not the individual end user, but other companies and services that rely on their auth). They should have teams monitoring for ddos or abuse and block them for all of their customers. OTP bombing is annoying but shouldn't lock out your customer unless you're doing IP binding, which I would personal avoid implementing except at a ASN or broad geolocation perspective, while allowing previously used IPs. Specifics here depend on threat model - is this more likely to be abused by remote criminals or by users' neighbors or colleagues. I mostly don't like them cause I tether my phone a lot and people's offices can use multiple egress IPs. Yes, magic links and OTPs should have expirations, prob around 5-10 mins depending on your threat model. And must be bound to specific users. In the same time period, I would limit to 3 emails. Also to note, unless you're using passkeys or Fido, otps of any kind (email, sms, totp app) are phishable.

email OTP shouldnt be a thing imo, go with legit 2FA or your just offloading your entire auth to the persons personal, phishable, email account, the security of which you have no control over.

So, the unfortunate problem is that secure email OTP can't be a thing until we have secure email in the first place. The additional problems your AI came up with are true, but pale in comparison to the first issue. It's certainly better than the nothing that most sites do (no MFA at all), but just barely. Going with SMS or Google Authenticator or some other ubiquitous OTP app would be a better idea, and also a lot less of an overall headache for you.

Don't try to implement security measures yourself. This is almost as bad as trying to write your own cryptography algorithms. Use something off-the-shelf.

Well, if authentication fall back to email as a recovery process then a password is just an accelerator on that process (due to the inherent best efforts delivery of email). Some low value sites offer email as a primary authentication route & I have no problems with this provided the value of any breached data is De-Minimums. In One Specific Way this method might actually be better that having a password, in that if the service stores no authentication secrets a breach of same for or with credential stuffing would be pointless. Getting such to work with low friction, reliability, & security is more of an ask, for which I would jokingly say Pick Two. One should certainly rate limit (per valid account) request because the email might take sometimes be delayed. One would like to have confirmation when requesting that the flow is working from the services POV, but that leaks information in ways that attackers can use. My suggestion on the protection front is to rate limit & log requests, then run a set of heuristic filters to detect the sorts of behaviours that need to be stopped.

Do you already have a forgot password flow that uses email? If so all of these issues already apply. Most of all, it sounds like you need DDoS/bot protection for the site as a whole. That should prevent users from getting flooded. Though you should also implement a rate limit on sending of emails in the event of delayed delivery and a real user continues to click resend. A safer method of enforcing yourself would be to block the IP of the offending requester across all authentication attempts rather than locking out a specific user. You would likely be better off using some IAM software to manage this rather than trying to implement it yourself, but it does still mean running (open-source, self-hosted) or paying for another service. Customer IAM tends to bill on active users so the cost should be on the low side if users don't access the site often.

Just implement an OIDC compliant system instead. Don't try to roll your own auth as it increases rapidly in complexity as you're finding out. If you find you still want to control the accounts yourself, there's self-hosted systems like Authentik, Authelia, or KeyCloak that will let you implement this auth flow and other OIDC providers.