how can I ensure that only 1 node executes a job in a distributed system?

Simple solution is to pull from a message queue. More advanced solution would be an actor system like Akka or Orleans. Start with the simplest solution that meets your needs, grow from there.

Pull from a message queue when an events queue is right there seems unecessary

I made no statements of implementation. Leveraging queues from postgres is a perfectly good solution

setnx works right up until a worker stalls on a gc pause or the box gets swapped out, the ttl expires, another node grabs the lock and now two of them are running the same job. thats basically the whole 'redlock is broken' argument, without a fencing token you cant actually prove you still hold it. since his durable state is already in postgres the skip locked route sidesteps all of it, the transaction itself is the lock

What's the reason for dropping k8s cron? Another option is to just run a single pod for running cron tasks (which is what you'll get with k8s cron anyway, just more flexible and easier to scale if necessary). User defined jobs is a task queue problem. Given that you're on Python, Celery offers both task queue and cron tasks, and have a page about locks: https://docs.celeryq.dev/en/main/tutorials/task-cookbook.html

Might be wild idea but if you can expose an endpoint and hit it from outside, it will be handled by single node, start jobs there.

Message queue would be a longer term solution but short term you could log that the job is executing in the dB and what node has picked it up. Then if other nodes try to kick in, check the dB....if it's already executing ...do nothing

You can use redis for that and use a redis lock/distributed lock around the job: https://redis.io/glossary/redis-lock/

Use a message queue like RabbitMQ Or BullMQ to queue the cron jobs needed and let the pods pick a job(acquiring a lock). This way you don't want to worry about scale and avoid duplicate executions of a cron job.

Celery

advisory locks are perfect for the hardcoded jobs, but think about the user defined jobs plan before you lock it in. a single pg_advisory_lock is basically one big global lock, totally fine with 5 known jobs but it gets awkward when users can spin up hundreds. the pattern that grows better is a jobs table and SELECT ... FOR UPDATE SKIP LOCKED, every worker just pulls the next unlocked row so you get real concurrency and single execution per job at the same time. and you already have postgres so theres no reason to drag redis in for locking, redlock is genuinely easy to get subtly wrong under a network partition

Kind of a hacky workaround, but if there is implicit auto-scaling, see what the RAM and throughput limits are that trigger it, then program the backend to stay under those.

Use a distributed lock. Redis with SETNX works for most cases. Acquire the lock before the job runs, release it after. If another node already holds it, skip. For Postgres you can also use SELECT FOR UPDATE SKIP LOCKED on a job-queue row. Both are simpler than bringing in a full task framework.

if you already have Postgres, i'd make Postgres the authority for this instead of adding another moving part. for a worker queue, use a jobs table and claim rows with select ... for update skip locked inside a transaction. each node can poll, but only one transaction gets the row. mark it running with a lease/expires_at so another node can recover it if the worker dies. for a scheduled job, create a job_runs table with a unique key like job_name + scheduled_minute. every node can try to insert that row; the one that wins the unique constraint runs it, the others do nothing. redis locks can work, but they need ttl/renewal/fencing handled carefully. since your durable state is already in postgres, keeping the lock next to the data is usually simpler to reason about.

yeah exactly, skip locked is the move when your state is already in postgres. redlock adds a whole distributed coordination problem that you just dont need if your work is already living in a transactional db. the fencing token argument is the real nail in the coffin though, without that guarantee you are just hoping the ttl window holds.

These are always fun architectural decisions to make for growing platforms props to you first of all for thinking through it and not just jumping to the first idea that makes sense. Distributed locks could solve your problem today but they aren't really the right tool for the job. As others have mentioned this is really a task queue problem normally. Something like celery (celerybeat offers scheduling) is a great solution, there are other similar task queues as well. One other note, it sounds like you might be trying to solve too many different problems with a single solution. Database cleanup might make more sense to do at the database level (think pg_cron if youre using postgres). The user defined jobs you mentioned could work nicely with celery, hard to say without full details. Happy to help out more if you'd like just shoot me a DM! Genuinely enjoy this type of architectural problem! Edit: Not that you asked about this but you said designed for k8s because of "expected very high load" which raises questions. What is "high load"? In my experience many people reach for k8s because they think it will solve their scaling and fault tolerance but they do so prematurely and end up with more to deal with down the road. Its great you designed it to work on k8s but do you need that right now? Your original question gets much easier with a VPS deployment (and you can still achive solid scale). Add a job queue on the side and you can scale that part independently and keep your monolith.