Like lots of things in life, not all questions have a hard & set answer. Nearly all composite materials containing flexing agent chemicals, (tires, hose, tubing, plastics, polymers, etc., etc.) degrade over time. Subjected to natural occurring degradation of exposure to ozone, UV light, and cold/heat cycles. There's this term called "plasticizing." It is kind of a term beyond my understanding as it seems to be used to describe how composites are formulated for flexibility (in a positive way) and also used as a negative term to describe how certain composites degrade over time.
One of my (too many) hobbies is old cars. In the car hobby, we collect and cherish NOS (New Old Stock) parts. I have had some old plastic parts literally crumble in my hands due to the flexing agents outgassing and the parts cracking (shrinking) as the flexing agents vaporize to the atmosphere over time. A good example is vintage steering wheels, vinyl seat covers, dash pads, etc. I have some unheated utility buildings where some of my parts are stored. For NOS parts I want to protect, I keep them in my home where the environment is better controlled for heat, humidity, UV, and ozone exposure. Those items seem to survive much better than the ones in the utility buildings.
So, in my opinion, as long as you keep your helmet stored in a controlled environment, (not in your barn, unheated garage, or in the sunlight behind a window) it should last longer. If you are a casual rider (like me) and bring your helmet inside after a ride, I think it should last much longer than a daily rider, with an open garage/barn, who daily hangs his helmet outside, on his handlebars, in the hot sun every day.
So...my answer to the original question is that it is a judgment call. Probably not what you are looking for, but I would also suggest that you buy a new helmet when you become uncomfortable and lose confidence in the one you are using. The peace of mind is worth the expense.