|
Examples
Example usage
Featured Simple ExampleMemcachedClient c=new MemcachedClient(
new InetSocketAddress("hostname", portNum));
// Store a value (async) for one hour
c.set("someKey", 3600, someObject);
// Retrieve a value (synchronously).
Object myObject=c.get("someKey");Taking Advantage of Asynchronous GetsMemcachedClient may be processing a great deal of asynchronous messages or possibly dealing with an unreachable memcached, which may delay processing. If a memcached is disabled, for example, MemcachedConnection will continue to attempt to reconnect and replay pending operations until it comes back up. To prevent this from causing your application to hang, you can use one of the asynchronous mechanisms to time out a request and cancel the operation to the server. // Get a memcached client connected to several servers
MemcachedClient c=new MemcachedClient(
AddrUtil.getAddresses("server1:11211 server2:11211"));
// Try to get a value, for up to 5 seconds, and cancel if it doesn't return
Object myObj=null;
Future<Object> f=c.asyncGet("someKey");
try {
myObj=f.get(5, TimeUnit.SECONDS);
} catch(TimeoutException e) {
// Since we don't need this, go ahead and cancel the operation. This
// is not strictly necessary, but it'll save some work on the server.
f.cancel(false);
// Do other timeout related stuff
}Establishing ConnectionsEstablishing a Binary Protocol ConnectionAs of release 2.0, the new binary wire protocol for memcached is supported. Below is an example of how you might connect to a server using the binary protocol: // Get a memcached client connected to several servers with the binary protocol
MemcachedClient c=new MemcachedClient(
new BinaryConnectionFactory(),
AddrUtil.getAddresses("server1:11212 server2:11212"));
// Proceed to do cool stuff with memcached using the binary protocol.Establishing a Binary Protocol SASL Connection// Create an AuthDescriptor, this one is for plain SASL, so the
// username and passwords are just Strings
AuthDescriptor ad = new AuthDescriptor(new String[]{"PLAIN"},
new PlainCallbackHandler(username, password));
// Then connect using the ConnectionFactoryBuilder. Binary is required.
try {
if (mc == null) {
mc = new MemcachedClient(
new ConnectionFactoryBuilder().setProtocol(Protocol.BINARY)
.setAuthDescriptor(ad)
.build(),
AddrUtil.getAddresses(host));
}
} catch (IOException ex) {
System.err.println("Couldn't create a connection, bailing out: \nIOException " + ex.getMessage());
}Establishing a Membase ConnectionYour exception handling should be different (and your class probably isn't Main), but here's a rough example of how to establish a connection to Membase. MemcachedClient mc;
try {
URI base = new URI("http://localhost:8091/pools");
ArrayList baseURIs = new ArrayList();
baseURIs.add(base);
mc = new MemcachedClient(baseURIs, "bucket_name", "bucket_password");
/* this could also be MemcachedClient(serverList, "default", "") in the case
* you're using a default bucket
*/
} catch (IOException ex) {
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
} catch (ConfigurationException ex) {
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
} catch (URISyntaxException ex) {
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
}
mc.set("hello", 0, "world");
String result = (String) mc.get("hello");
assert (result.equals("world"));
mc.shutdown(3, TimeUnit.SECONDS);
Using CASThe low-level CAS operation is a bit difficult to get right, so I've provided an abstraction that will allow one to more easily use it. Here's an example of how you might write a function that maintains a list of the 10 most recent Items using CAS (where Item is some kind of object that is meaningful within your application). Every call to addAnItem will place the new Item at the beginning of a List and store it (using CAS) in memcached. The important part to focus on here is the interaction between the "mutation" (a functional construct that describes how to transform an existing value into the desired one) and the client. Essentially, you only have to think of two things:
Note that the mutation may occur multiple times (up to 8,192 by default) when there's contention. In general, you shouldn't care about the retries and just think about what you want to do when you find a value you like. This function returns the newly manipulated and stored list. public List<Item> addAnItem(final Item newItem) throws Exception {
// This is how we modify a list when we find one in the cache.
CASMutation<List<Item>> mutation = new CASMutation<List<Item>>() {
// This is only invoked when a value actually exists.
public List<Item> getNewValue(List<Item> current) {
// Not strictly necessary if you specify the storage as
// LinkedList (our initial value isn't), but I like to keep
// things functional anyway, so I'm going to copy this list
// first.
LinkedList<Item> ll = new LinkedList<Item>(current);
// If the list is already "full", pop one off the end.
if(ll.size() > 10) {
ll.removeLast();
}
// Add mine first.
ll.addFirst(newItem);
return ll;
}
};
// The initial value -- only used when there's no list stored under
// the key.
List<Item> initialValue=Collections.singletonList(newItem);
// The mutator who'll do all the low-level stuff.
CASMutator<List<Item>> mutator = new CASMutator<List<Item>>(client, transcoder);
// This returns whatever value was successfully stored within the
// cache -- either the initial list as above, or a mutated existing
// one
return mutator.cas("myKey", initialValue, 0, mutation);
}
|
► Sign in to add a comment
The classname should be MemCachedClient?, not MemcachedClient?.
Greetings.. from where can i get a full set examples for this project.. I can see a lot of classes and java doc in this project.. but a good example set can be more helpful..
The asynchronous method is just great! Its performance is almost 80% that of the RAM hash table. I've tried the following out on my computer
The result is: AntClient? time/value:9140/10000:1.0940919037199124 AntClient? time/value:7532/10000:1.327668613913967 AntClient? time/value:8937/10000:1.1189437171310284
Here's a hint I had to figure out: When using the incr(...) command, though you get a long back from that method, the value is stored in a way that is interpreted as a String when calling get(...) from the same key.
I had originally tried to write a Long value to the key, then increment. This ended up returning an unexpected value.
@nelz Yeah, the incr stuff is just painful. I added the with-default methods to make it a bit easier to understand, but I could never make that one detail go away on the server.
hey.I couldn't find any place to set the weight of cache server. And I couldn't set the properties of socketIO.are there other solutions?
How should I use Transcoder interface? why do I need it? Who can give me an example? thanks!! who can answer my question?(I'm a beginner.)
wondering if there is anyway for MemcachedClient? to add a runtime added memcached server, I can restart MemcachedClient? to include the newly added server, but I don't think this is a good way. Use case is like: I started 3 memcached servers in the beginning, and I find that traffic is getting higher and I need to add more memcached server, how can this be communicated to MemcacheClient?? I just did not see an easy to achieve this.
dreamer:
The Transcoder interface is used to specify how objects turn into byte arrays and flag sets for storage within memcached. The default one mostly does java serialization with some special encoding for some native types (dates, for example).
If you don't know what it is, you probably don't need it. :)
liuyaping2005:
Adding support for reconfiguring a MemcachedClient? as you're describing would make the whole thing generally slower and much more complicated. Your best bet is to just build a new one and hand it off to various parts of your application.
Dustin:
thanks for the quick response. I think you mean to destroy the original MemcachedClient? and create another new MemcachedClient?, please let me know if that's not the case. Suppose it's the case, then I don't think it's an efficient one though, for example, I have 100 existing socket channels, and I added a new memcached server, meaning that I need to destroy existing 100 channels and recreate another 100 channels all over again in order to accommodate the new memcached server? this is just for one machine, if I have many such machines that each of them contains a MemcachedClient?, then each machine needs to do this also. On the other hand, adding more memcached servers at runtime will invalidate the whole caching structure, meaning that old caches will become invalid and everything will be cached again which will definitely hit performance. This will mean that if we need to add more memcached servers, it's better to restart all memcached servers and let the cache rebuilt again which is very expensive.
Please let me know if my understanding is wrong. Does this mean that it's the limitation of memcache or we did not get a good enough hashing algorithm yet?
liuyaping2005,
Occasionally shutting down and setting up a new client is pretty cheap. You're not going to do it enough to consider it an issue.
However, if you want to be able to dynamically one, every requests would have to deal with locks and dump all thread caches just in case a server was added or removed.
Slowing everything down to check for effects of inter-thread mutation will definitely make it slower and with more code, likely complicated code it's hard to undesrand. :)
Dsallings, Thanks for response. Another question, I have tried to save a custom serialized object in to memcached, and could successfully get it out. Why should I use Transcoder interface? It seems that I don't neet to implements the serialized interface If I use Transcoder interface. I feel using Transcoder is better than not, because it return the exactly object instead just one Object. Thanks again.
How are String objects put in the memcached? Are they serialized or written in plain/text format using a certain character encoding? I want to be able to use same memcached servers from different type of clients. (spymemcached on java, and from python/php etc). I am thinking of storing cached objects in a format like json in utf8 encoding. Should I write my own transcoders for reading and writing the objects in utf8 encoding?
By default, strings are encoded as UTF-8 and set with a generic kind of flag. That is, they are not stored in any java-specific way, and you should be able to verify that with tags.
However, they will be compressed by default if they're large enough. You can adjust this threshold when you create your transcoder.
Dreamer:
If you can't get the object back out, it may be due to classloader scope. The object is encoded by the requester in its own classloader, but will be decoded in a different scope.
This can happen if, for example, you use the cache in a .war, but the objects you're caching are in a inferior classloader.
Hi Dustin,
Regarding your previous comment about dynamically adding new memcached servers at runtime, I though the ketama hash algorithm should be able to provide a consistent hashing strategy such that when you add a new server, only a subset of the cache gets invalidated? I noticed in the source, there is already a KetamaConnectionFactory?. Does that provide consistent hashing?
The idea is that you cannot mutate an existing connection object (unleashes all kinds of performance and correctness problems).
You can bring up a new one with a new set of servers to replace your old one in a consistent hash and the properties of ketama hold.
Hi Dustin,
Another question. When using Ketama, does the implementation convert my key to a ketamahash? Or should I first convert my string key to a ketama hash and then use that for the set method?
ketama is just a locator. Use what keys you want. If the right thing doesn't happen, it's a bug. :)
Thanks,
Is there any restriction as to what type of objects I can set to the cache? Can I set a java Vector into the cache?
haroldvinson:
By default, anything serializable. You can always create your own transcoder if you'd like to be able to store custom, non-serializable objects.
Try it and see what works for you.
I didn't understand if is possible to set this client for working with persistent/pooled connection to memcached. In the company we've been experiencing a lot of connections to our nodes, and my application use just one singleton client... Since the aplication is replicated in 2 servers, I'd expect just 2 connections, can you give me any hint please? Thanks in advance!
Simone,
Yes, you can (and most likely should) use one connection from as many threads as you want throughout the lifetime of your application.
If you're seeing more than two connections, you're probably constructing more than two clients.
sorry for being dense - what does 'CAS' stand for/mean?
james:
It's modeled after concurrency primitives found in processors: http://en.wikipedia.org/wiki/Compare-and-swap
thanks for filling me in
Is there anyway to get a list of all keys in the cache? We have a need to be able to see what's currently in the cache.
JStrike, there is no such functionality. Most of the time when people have asked for it, it wouldn't actually be beneficial to solving their problems. What problem are you perceiving as this helping?
I'm considering using memcached to replace some caching we do in singletons. One of the things we have is a page where the admins can look at the data in the cache. Memcached is a bit of a paradigm shift and I'm just trying to work through the way we use singletons and see if it matches up well with how memcached works.
Looking at data in the cache isn't a use case, though. Cache is useful when you need it. The data you would see in a list isn't guaranteed to be there when you go back to fetch it.
Is the MemcachedClient? thread safe? Ie: can i just create one instance and let all my threads use this ??
@hoerup1981
Yes, it is expected that you will create one (or a very small number of) client and use it from many threads.
what is the default key expire time value .
Hi dstallings, Is there any plan to support "Multi-Set" mentioned at http://code.google.com/p/memcached/wiki/NewCommonFeatures ?
Multi-set escalation is automatic: http://dustin.github.com/2009/09/23/spymemcached-optimizations.html (though there's a bulk loader API to help it as well).
1. Does spymemcached support connection pool? If so, any published example of using spymemcached connection pool? 2. Is the default using consistent hashing? 3. Some post mentioned that spymemcached has problem auto reconnect to memcached servers. Is this still an issue?
I am trying to find a way to check if the memcachedclient is still alive or not but it seems to be there is no public api for me to do so. Can someone clarify this?
Thanks. Chad
I have a question. How I can check is my client connected or not on server. I have application where user can choose does he wants caching or no. If this value is true app will try to connect on memcache server
try {
Thanks in advance.
mom.
How does the atomic op work?
I don't think it supports arbitrary Java code, does it? Does it download the entire value to the client and then put the value back?
How about simpler CAS int and long?
Yes, it's arbitrary java code. It fetches the value, performs your transformation and then puts it back. If something changes the data in the middle of this process (between when it fetched it and tried to set a new value), it will start over.
It has to fetch it again when it changes, because all it knows is that it changed.
If it didn't change, it knows that, too, and will only have pulled it down the first time.
Hi.
I have a slightly different business case. I have multiple customers needing their data in high speed storage. I limit the amount of data each customer is allowed to store with me by setting each client up in its own pool. This way one customer can't push out other customer's data.
I'm looking to do an asynchronous high speed get across multiple memcached pools. Has anyone done this already? I don't want to duplicate effort if it has already been done.
Thanks. Dave
Hello.
How about callback-based asynchronous calls?
The problem with Future is that the calling thread needs polling and/or blocking to get access to the result.
With callbacks the calling thread could just forget the whole thing after the call, and the callback function (called by a memcached thread) would then continue when the data is available.
In my opinion callback-based parallelism is even cleaner than Future-based, and definitely "more parallel".
There is the MemcachedClient?.addOp method, but it's not public. We hacked some code in the net.spy.memcached package and were able to get callbacks for the get operation. But there was nothing similar for cas mutator. The CASOperation was too low-level for our needs and skills.
I get an error when i try to run the example which does this
Compiles,but there is a run time error
Exception in thread "main" java.lang.NoClassDefFoundError?: org/codehaus/jettison/json/JSONException
Caused by: java.lang.ClassNotFoundException?: org.codehaus.jettison.json.JSONExceptionYou need to add a few jars as dependencies if using Membase. Jettison, Netty and Apache Commons codec.
why i need to use CASMutation when i can use gets(String key, Transcoder<T> tc) and cas(String key, long casId, int exp, T value,Transcoder<T> tc) for synchronous operations?
Thks.
java.lang.NullPointerException?
I get above error when create Memcache client
client = new MemcachedClient?(baseURIs, "default", EMPTY_STRING);
@ra...@appsavvy.com,
The reason why you are seeing the error is because the Memcache client is setting the Http Basic Auth header when it doesn't need to if you are just using the default membase server setup (the membase server throws back a http 500 exception -> I see this in the wireshark trace). To get around this try:
client = new MemcachedClient??(baseURIs, "default", / username /null, EMPTY_STRING);
This stops the Basic Auth header from being set.
Any hope you would upgrade to netty 3.2.x soon? Unfortunately the current version of spymemcached will not work with the later version of netty because a method signature changed in HttpMethod???:
3.1.5 method
void setHeader(String name, String value);
3.2.4 method
void setHeader(String name, Object value);
Also any possibility spymemcached could connect to the membase servers via SSL?
Hi, I have posted sample example using spymemcached here: http://brainfry.in/programming/memcached-java-examples/
-Dmemcached.servers="localhost:11211 localhost:11212 localhost:11213 localhost:11214"
private static final String MEMCACHED_SERVERS = System.getProperty("memcached.servers");
public static MemcachedClient? getInstance() {
Ex.: Memcached.getInstance().set("channel", 3600 24, channelMap);
WARN 21 Set 2011 11:52:26 WARN MemcachedConnection?:? - Could not redistribute to another node, retrying primary node for channel
WHY?????????
Hello, I am new in spymemcached. Can we create our own custom hash algorithm and use is when set or get data? any one got tips for me?
Hello,
I have a curiosity of "Using a vBucket Aware Client" in http://docs.couchbase.org/membase-manual-1.7/membase-administration.html#idp140329436097312
I want to know the setting of vBucket Aware Client.
In my spring context,
<beans:bean id="memcachedClient" class="net.spy.memcached.spring.MemcachedClientFactoryBean"> <beans:property name="servers" value="1.2.3.4:11211 1.2.3.5:11211" /> <beans:property name="protocol" value="BINARY" /> <beans:property name="transcoder"> <beans:bean class="com.*****.KryoTranscoder" /> </beans:property> <beans:property name="opTimeout" value="1000" /> <beans:property name="timeoutExceptionThreshold" value="1998" /> <beans:property name="hashAlg" value="KETAMA_HASH" /> <beans:property name="locatorType" value="CONSISTENT" /> <beans:property name="failureMode" value="Redistribute" /> <beans:property name="useNagleAlgorithm" value="false" /> <beans:property name="authDescriptor"> <beans:bean class="net.spy.memcached.auth.AuthDescriptor"> <beans:constructor-arg index="0"> <beans:value>PLAIN</beans:value> </beans:constructor-arg> <beans:constructor-arg index="1"> <beans:bean class="net.spy.memcached.auth.PlainCallbackHandler"> <beans:constructor-arg index="0"> <beans:value>bucketName</beans:value> </beans:constructor-arg> <beans:constructor-arg index="1"> <beans:value>bucketNamePwd</beans:value> </beans:constructor-arg> </beans:bean> </beans:constructor-arg> </beans:bean> </beans:property> </beans:bean>It works fine. However, I wonder that this setting can handle the "vBucket Aware Client".
and one more,
is this spring beans able to connection pooling service or do I need to implement other way? such as common-pools or using Moxi ?
Thanks for your help, if you leave a comment about that.
I have an environment in which a libmemcached (c/c++ ) client has to read cached items from another spymemcached client.
Anyone has experiences with such a scenario? What kind of configurations can be used to make libmemcached to worked with spymemcached?
I have modified MemcacheClient? with the following method to allow asynchronous gets without having to poll a Future to get the result. I do not know how to efficiently poll thousands of Futures efficiently, so I wanted to use the asynchronous nature of the implementation directly. It appears to work:
My implementation of GetOperation?.Callback does all the work of keeping track of the state of the request, and another wrapper implements a timeout facility for all pending requests. Is this of general interest?