Memcache not freeing memory when a key expires

[Stephen Corgiat]

Hello,

We're seeing some weird behavior from memcached.. Hopefully somebody can
provide some knowledge on this issue.

When we set a key in memcache with a TTL of, say, 60 seconds, after 60
seconds has gone by the data "expires". This is normal, but it seems the
data is not truely freed by memcached (based on the 'bytes' stats field).
We've found that the only way to get the memory released is to try to do a
GET on the key.

I always thought if you had a bunch of keys with a TTL of 0, once all
available memory was consumed, memcache would start removing the least used
keys to make room for new keys. In our tests, this never happened.

We've tried installations on FreeBSD & Linux, 1.1.12 & 1.1.11... all with
the same results.

Thanks!
Stephen

[Fabian Thylmann]

Hi Stephen,

the overhead to keep track of expired keys and remove them from memory
is too large and is not needed. Once the key is accessed memcached will
see its expired and remove it from memory then. It will also simply
treat it as if its not there if the same key is set again.

How exactly memcached finds which key to remove when memory is all used
up I am not 100% sure myself.

Hope this helps a bit,
Fabian

Post by Stephen Corgiat
Hello,
We're seeing some weird behavior from memcached.. Hopefully somebody
can provide some knowledge on this issue.
When we set a key in memcache with a TTL of, say, 60 seconds, after 60
seconds has gone by the data "expires". This is normal, but it seems
the data is not truely freed by memcached (based on the 'bytes' stats
field). We've found that the only way to get the memory released is
to try to do a GET on the key.
I always thought if you had a bunch of keys with a TTL of 0, once all
available memory was consumed, memcache would start removing the least
used keys to make room for new keys. In our tests, this never happened.
We've tried installations on FreeBSD & Linux, 1.1.12 & 1.1.11... all
with the same results.
Thanks!
Stephen

[Brad Fitzpatrick]

As designed. It expires things lazily as they're accessed or fall off the
LRU. It's designed to be all O(1), so things like trees/priority queues
for finding expired items to expire them ahead of time violate that.

Post by Stephen Corgiat
Hello,
We're seeing some weird behavior from memcached.. Hopefully somebody can
provide some knowledge on this issue.
When we set a key in memcache with a TTL of, say, 60 seconds, after 60
seconds has gone by the data "expires". This is normal, but it seems the
data is not truely freed by memcached (based on the 'bytes' stats field).
We've found that the only way to get the memory released is to try to do a
GET on the key.
I always thought if you had a bunch of keys with a TTL of 0, once all
available memory was consumed, memcache would start removing the least used
keys to make room for new keys. In our tests, this never happened.
We've tried installations on FreeBSD & Linux, 1.1.12 & 1.1.11... all with
the same results.
Thanks!
Stephen

[Stephen Corgiat]

Thanks for your response..

Good to know if this was really the way it was supposed to work, hopefully
someone else finds this if they're wondering the same thing.

Since we love memcached so much and would like to continue using it for our
site, we decided to code in a "reaper" command which deletes keys that have
expired. We're testing it out and so far so good.

- Stephen

Post by Brad Fitzpatrick
As designed. It expires things lazily as they're accessed or fall off the
LRU. It's designed to be all O(1), so things like trees/priority queues
for finding expired items to expire them ahead of time violate that.

Post by Stephen Corgiat
Hello,
We're seeing some weird behavior from memcached.. Hopefully somebody can
provide some knowledge on this issue.
When we set a key in memcache with a TTL of, say, 60 seconds, after 60
seconds has gone by the data "expires". This is normal, but it seems

the

Post by Stephen Corgiat
data is not truely freed by memcached (based on the 'bytes' stats

field).

Post by Stephen Corgiat
We've found that the only way to get the memory released is to try to do

a

Post by Stephen Corgiat
GET on the key.
I always thought if you had a bunch of keys with a TTL of 0, once all
available memory was consumed, memcache would start removing the least

used

Post by Stephen Corgiat
keys to make room for new keys. In our tests, this never happened.
We've tried installations on FreeBSD & Linux, 1.1.12 & 1.1.11... all

with

Post by Stephen Corgiat
the same results.
Thanks!
Stephen

[Jason Rimmer]

I'm on the list digest so my apologies if this has already been addressed:
Why would you do this; what purpose would the reaper serve? As Brad
mentioned previously memcached is an LRU cache with essentially a lazy
'reaper' to maintain O(1). In general an active reaper would have two
purposes: 1) Remove expired entries to make room for more and 2) Ensure
that expired entries are no longer accessible. Brad addressed both
points in that for the first case expired entries are automatically
removed via LRU when memcached comes under memory pressure. With the
second case memcached ensures that expired entries are not retrievable
(the fact that those entries happen to be removed or 'reaped' when
accessed is orthogonal to the retrieval attempt).
What are you trying to accomplish with your active reaper and how will
the reaper know that a key is expired and hence appropriate for 'removal'?

Date: Mon, 27 Feb 2006 18:27:16 -0800
Subject: Re: Memcache not freeing memory when a key expires
Content-Type: text/plain; charset="iso-8859-1"
Thanks for your response..
Good to know if this was really the way it was supposed to work,
hopefully someone else finds this if they're wondering the same thing.
Since we love memcached so much and would like to continue using it
for our site, we decided to code in a "reaper" command which deletes
keys that have
expired. We're testing it out and so far so good.
- Stephen

[Evan Fribourg]

Under our unit testing with memcached, when we insert an item with a TTL
greater than 0, when the item expires the item is not removed from memory
until a GET command is issued on the key. Based on the lazy reaper, this is
normal operating procedure. In production, however, our memcached server
will max out its usable space and NOT seem to drop keys based on the LRU
scheme, with settings.evict_to_free == 1.

The keys that we use in our server are mostly unique -- they tend be to be
re-used for a little while, and then not used ever again. Because there was
no reaper to go through and delete these expired keys, the memcached server
kept growing and growing until we reached a point where memcache was unable
to evict any items. Our implementation of a reaper command within memcached
collects our garbage and enables us to not have to restart memcached every
few days.

Evan Fribourg


-----Original Message-----
From: memcached-***@lists.danga.com
[mailto:memcached-***@lists.danga.com] On Behalf Of Jason Rimmer
Sent: Tuesday, February 28, 2006 7:18 AM
To: ***@lists.danga.com
Subject: Re: Memcache not freeing memory when a key expires

I'm on the list digest so my apologies if this has already been
addressed:
Why would you do this; what purpose would the reaper serve? As Brad
mentioned previously memcached is an LRU cache with essentially a lazy
'reaper' to maintain O(1). In general an active reaper would have two
purposes: 1) Remove expired entries to make room for more and 2) Ensure that
expired entries are no longer accessible. Brad addressed both points in
that for the first case expired entries are automatically removed via LRU
when memcached comes under memory pressure. With the second case memcached
ensures that expired entries are not retrievable (the fact that those
entries happen to be removed or 'reaped' when accessed is orthogonal to the
retrieval attempt).
What are you trying to accomplish with your active reaper and how
will the reaper know that a key is expired and hence appropriate for
'removal'?

[Peter van Dijk]

Post by Evan Fribourg
Under our unit testing with memcached, when we insert an item with a TTL
greater than 0, when the item expires the item is not removed from memory
until a GET command is issued on the key. Based on the lazy reaper, this is
normal operating procedure. In production, however, our memcached server
will max out its usable space and NOT seem to drop keys based on the LRU
scheme, with settings.evict_to_free == 1.

I'm putting 5 bucks on: your ulimits are set too low, so the -kernel-
is preventing memcache from allocating more memory; but, as memcache
hasn't reached the memory limit you configured -in memcache-, it is not
cleaning up yet.

Cheers,
Peter.