Okay, it’s actually more complex than that. Because on self managed nodes, kubernetes is not the only thing that’s running, so it can make sense to set limits for other non kubernetes workloads hosted on those nodes. And memory is a bit different from CPU. You will have to do some testing and YMMV but just keep the difference between requests and limits in mind.
In order for them not to be OOM Killed, you have to set the memory requests for them above their highest spike, which means most of the time they’re only using like 25% or so of their memory allocation.
Are you sure? Only limits should limit the total memory usage of a pod? Requests should happily let pods use more memory than the request size.
One thing I am curious about is if your pods actually need that much memory. I have heard (horror) stories, where people had an application in Kubernetes with a memory leak, so what they did instead of fixing the memory leak, was to just regularly kill pods and restart new ones that weren’t leaking yet. :/
To answer your actual question about memory optimization, no. Even google still “wastes” memory by having requests and limits higher than what pods usually use. It is very difficult to prune and be ultra efficient. If an outage due to OOM costs more than paying for more resources would, then people just resort to the latter.
Others are correct, the problem is the software. You are right to use memory requests and limits. The limits being the max it will use, but hopefully other pods won’t be using all of their limits at once.
So all of the pods’ memory requests on a given node will sum to < 100% of the total available memory. So you can of course say your pod requests the highest amount of ram it will ever need, but that does mean it’s reserved for that pod and won’t be used anywhere else even during downtime
K8s will allow over provisioning of ram for the limits though because it assumes it will not always need that as you are seeing.
What you can do is to set a priority class on the pod so when it spikes and you don’t have enough ram, it will kill some other pod instead of yours, but that makes other pods more volatile of course.
There’s many options at your disposal, you’ll have to decide what works best for your use case.
That seems like a problem of the application, no? If the workloads have memory leaks or are too eager to get memory to itself, then no cluster will be able to make it perform better.
Answering the easy dumb questions is now done by AI pretty good. I would say even better because the AI does not care about flagging your question for duplicate or mocking you for not being precise.
StackOverflow needs to refocus on those questions not answered by AI easily. They need to adjust their business to fewer questions that need thorough investigation and specialist solutions. That is a hard thing to do with only volunteers to answer these questions, so maybe they need to switch to a paid model which pays the correctly flagged answer a cut of the fee…
I never even thought of using Stack Overflow to cheat when doing my homework. Still heard of its reputation during college and decided I should never ask a question there (even now after I’ve graduated so it absolutely would not be homework help), lest someone, yes, be mean to me.
Well that might suck phenomenally because at least in stackoverflow people (albeit sometimes toxically) point out issues with replies/codes. In GitHub anyone can post any code that may have critical errors. Even worse GitHub wil be full of AI generated code like internet is filled with AI generated videos. So things may start to get weird when AI gets trained on AI code.
Yeah, either trained on or they do a web search when you ask them knowledge questions. Gonna be an interesting one, when all the actual knowledge sources have disappeared.
It was popular before the current ridiculous moderationlevels. Now people just go there for the older content, since all of the newer stuff gets closed as ‘duplicate’ despite not being a duplicate.
While I’m sure you can find examples of those, the reality is that mods are right the vast majority of the time. The truth people don’t want to acknowledge is that the strict moderation keeping up the quality of the answers is the reason stack overflow was so popular.
And to add on to your experience, I find I’m also too impatient to ask the internet for help. The harder and more annoying an issue is, the more motivated I am to finally solve it, meaning having to wait for someone to potentially answer, maybe the next day if ever, is not very appealing. I’d rather spend hours on the issue and solve it myself, which still means it’s solved faster.
How valuable will all of that code be without the context added by human conversation emphasizing and explaining aspects of code?
Sure, this will work for programming languages that are unchanging and dead, but as people begin to need to learn new aspects of programming language this system will spectacularly fail. AI does not beat high quality human conversations about a technical topic, it can only badly summarize them.
Beyond AI tools in IDEs, the reputation of the site’s attitude towards new question posters definitely doesn’t help. Neither does the site’s pivot towards AI that alienated a bunch of the power users. The reputation issue isn’t new though, and I’m not sure of when the AI pivot started.
I’m also not sure if it’s a great metric. Personally, anything complicated enough for me to want outside help with is usually so far into the specifics of my work environment that it’s not safe to ask about online. It’d either be too revealing or devolve into the classic “I refuse to answer your question because I refuse to believe your starting scenario and limitations”. Otherwise it tends to be stuff I can sort out by piecing together stuff from blogs, documentation, and tangentially related stackOverflow answers. Not always easy, but easy enough that I’d rather work through it myself instead of asking the internet for help.
I was a game developer who was interested in backend, automations and process optimizations. Eventually, I learned all the basic DevOps stuff to find a real DevOps job where I developed my skills further
So I fell ass backwards into DevOps. I have CS education and during covid I had to get a job as a SysAdmin. After I while I became the soft skills SysAdmin for the company I worked at. I would put together service contracts, be the point of contact, and coordinate between all parties involved. I still did technical tasks, I was just the dude that could make things happen. Customers liked to work with me more than the company owner.
I had to move and when I updated my LinkedIn, I was contacted for DevOps roles. Apparently I have the exact skill set required for being a DevOps developer. I believe that you can’t be a DevOps developer on purpose, unless you go to school twice.
Yes, in fact we are now at a point where people need to provide justification that something can’t be run in k8s before provisioning the infra to run their non-k8s workload. Aka, kubernetes is an enforced default in our shop.
PieFed.zip
https://home.robusta.dev/blog/stop-using-cpu-limits
Okay, it’s actually more complex than that. Because on self managed nodes, kubernetes is not the only thing that’s running, so it can make sense to set limits for other non kubernetes workloads hosted on those nodes. And memory is a bit different from CPU. You will have to do some testing and YMMV but just keep the difference between requests and limits in mind.
But my suggestion would be to try to see if you can get away with only setting requests, or with setting high very high limits. See: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/#if-you-do-not-specify-a-memory-limit
Are you sure? Only limits should limit the total memory usage of a pod? Requests should happily let pods use more memory than the request size.
One thing I am curious about is if your pods actually need that much memory. I have heard (horror) stories, where people had an application in Kubernetes with a memory leak, so what they did instead of fixing the memory leak, was to just regularly kill pods and restart new ones that weren’t leaking yet. :/
To answer your actual question about memory optimization, no. Even google still “wastes” memory by having requests and limits higher than what pods usually use. It is very difficult to prune and be ultra efficient. If an outage due to OOM costs more than paying for more resources would, then people just resort to the latter.
Others are correct, the problem is the software. You are right to use memory requests and limits. The limits being the max it will use, but hopefully other pods won’t be using all of their limits at once.
So all of the pods’ memory requests on a given node will sum to < 100% of the total available memory. So you can of course say your pod requests the highest amount of ram it will ever need, but that does mean it’s reserved for that pod and won’t be used anywhere else even during downtime
K8s will allow over provisioning of ram for the limits though because it assumes it will not always need that as you are seeing.
What you can do is to set a priority class on the pod so when it spikes and you don’t have enough ram, it will kill some other pod instead of yours, but that makes other pods more volatile of course.
There’s many options at your disposal, you’ll have to decide what works best for your use case.
That seems like a problem of the application, no? If the workloads have memory leaks or are too eager to get memory to itself, then no cluster will be able to make it perform better.
You can see the similarities. I really like the trust levels both systems use instead of the classic system of fixed user tiers.
Answering the easy dumb questions is now done by AI pretty good. I would say even better because the AI does not care about flagging your question for duplicate or mocking you for not being precise.
StackOverflow needs to refocus on those questions not answered by AI easily. They need to adjust their business to fewer questions that need thorough investigation and specialist solutions. That is a hard thing to do with only volunteers to answer these questions, so maybe they need to switch to a paid model which pays the correctly flagged answer a cut of the fee…
I never even thought of using Stack Overflow to cheat when doing my homework. Still heard of its reputation during college and decided I should never ask a question there (even now after I’ve graduated so it absolutely would not be homework help), lest someone, yes, be mean to me.
Well that might suck phenomenally because at least in stackoverflow people (albeit sometimes toxically) point out issues with replies/codes. In GitHub anyone can post any code that may have critical errors. Even worse GitHub wil be full of AI generated code like internet is filled with AI generated videos. So things may start to get weird when AI gets trained on AI code.
Yeah, either trained on or they do a web search when you ask them knowledge questions. Gonna be an interesting one, when all the actual knowledge sources have disappeared.
That’s a ridiculous assertion. The moderation and complaints were the same a decade ago.
Was so popular. Past tense.
It was popular before the current ridiculous moderationlevels. Now people just go there for the older content, since all of the newer stuff gets closed as ‘duplicate’ despite not being a duplicate.
While I’m sure you can find examples of those, the reality is that mods are right the vast majority of the time. The truth people don’t want to acknowledge is that the strict moderation keeping up the quality of the answers is the reason stack overflow was so popular.
Yeah, ok. Now explain these:
And to add on to your experience, I find I’m also too impatient to ask the internet for help. The harder and more annoying an issue is, the more motivated I am to finally solve it, meaning having to wait for someone to potentially answer, maybe the next day if ever, is not very appealing. I’d rather spend hours on the issue and solve it myself, which still means it’s solved faster.
Is that before or after all the closed due to duplicate?
Waaah I copy pasted my homework question and the expert that donates his time for free was mean to me
How valuable will all of that code be without the context added by human conversation emphasizing and explaining aspects of code?
Sure, this will work for programming languages that are unchanging and dead, but as people begin to need to learn new aspects of programming language this system will spectacularly fail. AI does not beat high quality human conversations about a technical topic, it can only badly summarize them.
Meanwhile, Discourse forum software continues to grow in popularity an doesn’t have the reputation for toxic communities.
Jeff Atwood co-founded both StackOverflow and Discourse.
Yeah, good luck fixing issues with “intelligence” based on MY code.
Well, they used to be trained on Stack Overflow.
In the future they’ll be trained on all your code, as per the end user licence agreement you clicked through.
Beyond AI tools in IDEs, the reputation of the site’s attitude towards new question posters definitely doesn’t help. Neither does the site’s pivot towards AI that alienated a bunch of the power users. The reputation issue isn’t new though, and I’m not sure of when the AI pivot started.
I’m also not sure if it’s a great metric. Personally, anything complicated enough for me to want outside help with is usually so far into the specifics of my work environment that it’s not safe to ask about online. It’d either be too revealing or devolve into the classic “I refuse to answer your question because I refuse to believe your starting scenario and limitations”. Otherwise it tends to be stuff I can sort out by piecing together stuff from blogs, documentation, and tangentially related stackOverflow answers. Not always easy, but easy enough that I’d rather work through it myself instead of asking the internet for help.
Question… what were all those AI tools in IDEs trained on?….
……….
I was a game developer who was interested in backend, automations and process optimizations. Eventually, I learned all the basic DevOps stuff to find a real DevOps job where I developed my skills further
Soft skills stand out, in the description of your old job make sure that there are soft skills tasks listed. You still need to be extremely technical.
LOL, same at my last job! Any tips on updating LinkedIn or job hunting?
So I fell ass backwards into DevOps. I have CS education and during covid I had to get a job as a SysAdmin. After I while I became the soft skills SysAdmin for the company I worked at. I would put together service contracts, be the point of contact, and coordinate between all parties involved. I still did technical tasks, I was just the dude that could make things happen. Customers liked to work with me more than the company owner.
I had to move and when I updated my LinkedIn, I was contacted for DevOps roles. Apparently I have the exact skill set required for being a DevOps developer. I believe that you can’t be a DevOps developer on purpose, unless you go to school twice.
I do!
You have a cat?
Well I am for sure 1/4 of the way there, and working on the rest.
Yes, in fact we are now at a point where people need to provide justification that something can’t be run in k8s before provisioning the infra to run their non-k8s workload. Aka, kubernetes is an enforced default in our shop.
Yes. $70/month is nothing compared to salaries, having a unified toolset saves time and energy.
We run everything in gke and use minikube on dev workstations to reproduce prod.