Cassandra on Random Ramblings

Getting the DataStax cass-operator working on Mac

Tue, 23 Feb 2021 00:00:00 +0000

Getting the DataStax cass-operator working on Mac

Just leaving this here for anyone coming back to this later. This works as of Feb 23 2021 on Big Sur

install docker desktop https://www.docker.com/products/docker-desktop
install k3d https://github.com/rancher/k3d#get you can just run curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
create cluster: k3d cluster create
install operator manifest kubectl create -f https://raw.githubusercontent.com/datastax/cass-operator/v1.5.1/docs/user/cass-operator-manifests-v1.19.yaml
download a cassandra data center yaml curl -O https://gist.githubusercontent.com/rsds143/0e9263ed4287d888fab36eb3e4aec502/raw/f919172c64452a2277b7523d680a9e12916d0d39/cassandra-dc.yaml
run kubectl create -f cassandra-dc.yaml --namespace cass-operator
wait for cassandra to come up kubectl logs cluster1-dc1-default-sts-0 --namespace cass-operator server-system-logger
get username export CQLUSER=$(kubectl get secret cluster1-superuser -o json -n cass-operator | grep \"username\" | sed -E 's/\"(.+)\": \"(.+)\"/\2/' | base64 -d)
get password export CQLPASS=$(kubectl get secret cluster1-superuser -o json -n cass-operator | grep \"password\" | sed -E 's/\"(.+)\": \"(.+)\"/\2/' | sed -E 's/,*$//g' | base64 -d )
log into cqlsh kubectl exec -it -n cass-operator cluster1-dc1-default-sts-0 -- cqlsh -u $CQLUSER -p $CQLPASS
enjoy your cassandra cluster

The yaml I have linked too above is here

Apache NiFi and Cassandra

Mon, 08 Feb 2021 00:00:00 +0000

Apache NiFi and Cassandra

Looked at Apache NiFi for the first time in a long time as someone didn’t know if it had support for stored procedures or not. I looked up the latest source and sure enough I see no stored procedures used for any of the statements. Looking at the PutCassandraRecord.java I see inserts and updates are just statements that are never prepared:

 query = generateInsert(cassandraTable, schema, recordContentMap);

and

Heap Size Around 32 GB

Wed, 03 Feb 2021 00:00:00 +0000

Heap Size Around 32 GB

note: originally written Feb 3 2021

There was a question this morning around a pretty common discussion point around 31 gb versus 32 gb of heap. This article discusses this in more detail, the key point follows:

Be careful when you increase your application heap size over 32G (from under 32G to over 32G) – JVM switches to 64 bit object references at that moment, which means that your application may end up with less available heap space. A rule of thumb is to jump from 32G right to 37-38G and continue adding memory from that point. The actual area of “grey” territory depends on your application – the bigger an average Java object in your application, the smaller is the overhead.

Starting Cassandra - Data modeling in the brief

Wed, 05 Feb 2020 00:00:00 +0000

Starting Cassandra - Data modeling in the brief

Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point.

Introduction

Cassandra distributes data around the cluster via the partition key.

CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));

In the above table the partition key is postal_code and the clustering column isid. The partition key will locate the data on the cluster for us. The clustering column allows us multiple rows per partition key so that we can filter how much data we read per partition. The ‘optimal’ query is one that retrieves data from only one node and not so much data that GC pressure or latency issues result. The following query is breaking that rule and retrieving 2 partitions at once via the IN parameter.

Starting Cassandra - Load testing Cassandra in brief

Tue, 04 Feb 2020 00:00:00 +0000

Starting Cassandra - Load testing Cassandra in brief

An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a lot of the nuance that one would normally get into when talking about load testing cassandra.

If you have no data model in mind

Use cassandra stress since it’s around:

first initialize the keyspace with RF3 cassandra-stress "write cl=ONE no-warmup -col size=FIXED(15000) -schema replication(strategy=SimpleStrategy,factor=3)"
second run stress cassandra-stress "mixed n=1000k cl=ONE -col size=FIXED(15000)
repeat as often as you’d like with as many clients as you want.

If you have a specific data model in mind

You can use cassandra-stress, but I suspect you’re going to find your data model isn’t supported (collections for example) or that you don’t have the required PHD to make it work the way you want. There are probably 2 dozen options from here you can use to build your load test, some of the more popular ones are gatling, jmeter, and tlp-stress. My personal favorite for this though, write a small simple python or java program that replicates your use case accurately in your own code, using a faker library to generate your data. This takes more time but you tend to have less surprises in production as it will accurately model your code.

Starting Cassandra - Setting up a Multi-DC environment

Mon, 03 Feb 2020 00:00:00 +0000

Starting Cassandra - Setting up a Multi-DC environment

This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers.

A new cluster

In cassandra.yaml set endpoint_snitch: GossipingPropertyFileSnitch, some prefer PropertyFileSnitch for the ease of pushing out one file. GossipingPropertyFileSnitch is harder to get wrong in my experience.
set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.
finish adding all of your nodes.
if using authentication, set system_auth keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};, run repair after changing RF
nodetool repair -pr system_auth on each node in the cluster on the new keyspace.
create your new keyspaces for your app with RF 3 in each dc (much like you did for the system_auth step above).
nodetool repair -pr whatever_new_keyspace on each node in the cluster on the new keyspace.

An existing cluster

This is harder and involves more work and more options, but I’m going to discuss the way that gets you into the least amount of trouble operationally.

Snakeyaml error handling is tough

Fri, 27 Dec 2019 09:30:00 -0600

"Snakeyaml error handling is tough"

Cassandra for years has relied on snakeyaml to parse configuration files, and it does a fine job, really not a lot of complaints. Until one day a colleague of mine discovered an issue where a permission issue incorrectly reported as an invalid yaml file (complete with the supposedly offending key). Digging into the source code and trying to match up the message. I found that we were handling YAMLException

Cassandra - Batch Loading Without the Batch The Nuanced Edition

Fri, 29 Apr 2016 16:03:56 +0000

Cassandra - Batch Loading Without the Batch The Nuanced Edition

My [previous post on this subject](https://blog.foundev.pro/2014/08/28/cassandra-batch-loading-without-the-batch-keyword.html) has proven extraordinarily popular and I get commentary on it all the time, most of it quite good. It has however, gotten a decent number of comments from people quibbling with the nuance of the post and pointing out it’s failings, which is fair because I didn’t explicitly spell this out as a “framework of thinking” blog post or a series of principles to consider. This is the tension between making something approachable and understandable to the new user but still technically correct for the advanced one. Because of where Cassandra was at the time and the user base I was encountering day to day, I took the approach of simplification for the sake of understanding. However, now I think is a good time to write this up with all the complexity and detail of a production implementation and the tradeoffs to consider.

TLDR

Find the ideal write size it can make a 10x difference in perf (10k-100k is common).
Limit threads in flight when writing.
Use tokenaware unlogged batches if you need to get to your ideal size.

Details on all this below.

Spark job that writes to Cassandra just hangs when one node goes down?

Tue, 05 Apr 2016 00:00:00 +0000

Spark job that writes to Cassandra just hangs when one node goes down?

### So this was hyper obvious once I saw the executor logs and the database schema, but this had me befuddled at first and the change in behavior with one node should have made it obvious. {#23da}

The code was simple, read a bunch of information, do some minor transformations and flush to Cassandra. This was nothing crazy. But during the users fault tolerance testing, the job would just seamingly hang indefinitely when a node was down.

Cassandra Query Patterns: Not using the “in” query for multiple partitions.

Mon, 22 Sep 2014 22:12:58 +0000

Cassandra Query Patterns: Not using the “in” query for multiple partitions.

So lets say you’re doing you’re best to data model all around one partition. You’ve done your homework and all you queries look like this:

 SELECT * FROM my_keyspace.users where id = 1

Over time as features are added however, you make some tradeoffs and need to start doing queries across partitions. At first there are only a few queries like this.

Cassandra - Batch loading without the Batch keyword

Thu, 28 Aug 2014 13:10:55 +0000

Cassandra - Batch loading without the Batch keyword

ATTENTION:

This post is intentionally simplistic to help explain tradeoffs that need to be made. If you are looking for some production level nuance go read this afterwords.

Batches in Cassandra are often mistaken as a performance optimization. They can be but only in rare cases. First we need to discuss the different types of batches:

Unlogged Batch

A good example of an unlogged batch follows and assumes a partition key of date. The following batch is effectively one insert because all inserts are sharing the same partition key. Assuming a partition key of date the above batch will only resolve to one write internally, no matter how many there are as long as they have the same date value. This is therefore the primary use case of an unlogged batch: