Caching is a technique used to store frequently accessed data in a temporary storage layer to improve system performance and reduce latency. There are various caching strategies, each with different use cases, benefits, and trade-offs.

Different caching strategies dictate how data is read from and written to the cache and the underlying data store (e.g., a database). Below, are the key caching types—Read-Through, Write-Through, and others like Write-Back, Write-Around, and Cache-Aside.

---

1. Read-Through Cache

In a read-through cache, the application requests data from the cache. If the data isn’t present (cache miss), the cache itself fetches it from the underlying data store, stores it, and returns it to the application. The application doesn’t directly interact with the data store for reads.

• How It Works:
  1. Application requests data from the cache.
  2. Cache checks for the data:
     • Hit: Returns data immediately.
     • Miss: Cache queries the data store, updates itself, then returns the data.

Pros

✔ Automatic Cache Population – Ensures that frequently accessed data is available in the cache.

✔ Consistent Read Patterns – Applications always query the cache first, reducing database load.

✔ Improved Performance – Faster data access due to lower latency.

Cons

❌ Higher Latency on Misses – Initial misses cause database queries, slowing performance.

❌ Stale Data Risk – If the database is updated, the cache may serve outdated data until refreshed.

• Example use cases:
  • User Profiles: Frequently accessed user data in applications like Facebook or Twitter.
  • Product Catalogs: E-commerce applications where product details are cached for quick retrieval.
  • CDN's: Used in content delivery networks (CDNs) like Cloudflare, where edge nodes fetch from origin servers on misses.

---

2. Write-Through Cache

In a write-through cache, every write operation from the application goes through the cache, which immediately updates both itself and the underlying data store synchronously.

• How It Works:
  1. Application writes data to the cache.
  2. Cache updates its own storage and synchronously writes to the data store.
  3. Write is acknowledged only after both are updated.

Pros

✔ Strong Consistency – Ensures that data in the cache is always up to date with the database.

✔ No Cache Miss Delays – Since data is written to the cache immediately, subsequent reads will be fast.

✔ Reduced Database Load – Reads are served directly from the cache.

Cons

❌ Slower Writes – Since every write updates both the cache and the database, latency increases.

❌ Unnecessary Caching of Less-Used Data – Even rarely accessed data is cached, increasing memory usage.

• Example use cases:
  • Financial Systems: Banking applications where data consistency is critical.
  • Session Management: Keeping user session data synchronized across distributed systems. 

---

3. Write-Back Cache (Write-Behind)

In a write-back cache, writes are made to the cache first, and updates to the underlying data store are deferred (asynchronously). This reduces write latency since the database is updated only after a delay.

• How It Works:
  1. Application writes to the cache.
  2. Cache acknowledges the write immediately.
  3. Cache later syncs the data to the data store (e.g., in batches or at intervals).

Pros

✔ Faster Writes – Writes are performed in memory first, reducing response time.

✔ Batch Writes – Multiple updates can be grouped into a single batch to optimize database writes.

✔ Improves Database Performance – Reduces the number of direct database writes.

Cons

❌ Risk of Data Loss – If the cache fails before persisting data to the database, recent writes may be lost.

❌ Complex Implementation – Requires mechanisms for handling failures and ensuring durability.

• Example use cases:
  • Logging Systems: High-throughput logging where logs are buffered before being written to storage.
  • Analytics: Clickstream data processing before persisting in a database.

---

4. Write-Around Cache

In a write-around cache, writes bypass the cache entirely and go directly to the underlying data store. The cache only stores frequently accessed data, avoiding unnecessary cache pollution.

• How It Works:
1. Application writes directly to the data store.
2. Cache isn’t updated during the write.
3. Future reads may trigger cache population (e.g., via read-through).

Pros

✔ Prevents Caching of Cold Data – Data that is written once and never read does not consume cache memory.

✔ Efficient Memory Usage – Only popular items remain in the cache.

Cons

❌ Cache Misses on Recent Writes – Since new writes are not cached, reading immediately after writing will result in a cache miss.

❌ Higher Read Latency – Applications relying heavily on cache hits may experience increased database queries. Not ideal for frequently updated, frequently read data.

• Example use cases:
• Content Delivery Networks (CDNs): Storing frequently accessed web assets while new content is retrieved from the origin server.
• Streaming Services: Caching popular video metadata but not every user-uploaded video.

---

5. Cache-Aside (Lazy Loading)

In a cache-aside strategy, the application is responsible for managing the cache. It explicitly checks the cache, fetches from the data store on a miss, and updates the cache manually.

• How It Works:
  1. Application checks cache for data:
     • Hit: Returns data.
     • Miss: Queries data store, updates cache, then returns data.
  2. Writes go to the data store, and the application decides whether to update or invalidate the cache.

Pros

✔ Efficient Memory Usage – Only frequently accessed data is stored in the cache.

✔ Reduces Stale Data – Data is fetched from the database only when required.

✔ Simple Implementation – Works well with existing applications.

Cons

❌ Cache Miss Penalty – Every cache miss results in a direct database query, increasing latency.

❌ No Automatic Cache Population – Data must be manually loaded into the cache.

• Example use cases:
  • Web Applications: Caching rendered HTML pages.
  • API Rate Limiting: Storing API responses to reduce backend load.

---

Comparison of Caching Strategies

Type	Read Latency	Write Latency	Consistency	Data Loss Risk	Complexity
Read-Through	High on miss	N/A	Strong	Low	Low (cache-side)
Write-Through	Low	High	Strong	Low	Moderate
Write-Back	Low	Low	Eventual	High	High
Write-Around	High on miss	Low	Weak	Low	Low
Cache-Aside	High on miss	Variable	Variable	Low	High (app-side)

---

Conclusion:

Choosing the right caching strategy depends on your use case:

• For fast reads with automatic caching → Use Read-Through.
• For strong consistency → Use Write-Through.
• For performance-optimized writes → Use Write-Behind.
• For application-controlled caching → Use Cache-Aside.
• For avoiding unnecessary caching → Use Write-Around.

Consistency Levels in Distributed Systems

In distributed systems, consistency defines the guarantees about the visibility and ordering of updates across different nodes. The choice of consistency level impacts performance, availability, and reliability.  

Understanding consistency levels helps designing the system effectively within given constraints.

---

1. Strong Consistency

• Definition: Strong consistency guarantees that every read operation returns the most recent write operation’s result, regardless of which node in a distributed system is accessed. All nodes see the same data at the same time.
• How It Works: After a write is acknowledged, all subsequent reads (from any client or node) reflect that write. This often requires synchronization mechanisms like locks or consensus protocols.
• Examples:
  • Relational databases with ACID transactions (e.g., PostgreSQL, MySQL with strict settings).
  • Distributed systems using two-phase commit (2PC).
  • Distributed Databases: Google Spanner, CockroachDB
• Advantages:
  • Predictable and intuitive behavior for applications (what you write is what you read immediately).
  • Ideal for systems requiring absolute data correctness, like financial transactions.
• Disadvantages:
  • High latency due to coordination overhead between nodes.
  • Reduced availability in the face of network partitions (per the CAP theorem, strong consistency often sacrifices availability).
• Use Case: Banking systems where account balances must always reflect the latest transactions.

---

2. Weak Consistency

• Definition: Weak consistency does not guarantee that a read operation will reflect the most recent write. Updates may propagate to nodes lazily, and clients might see stale or inconsistent data temporarily.
• How It Works: Nodes operate independently, and synchronization happens opportunistically (e.g., via gossip protocols or background replication). There’s no strict ordering of operations.
• Examples:
  • Early distributed systems with minimal coordination.
  • DNS (Domain Name System), where updates propagate slowly.
• Advantages:
  • High availability and low latency since operations don’t block for synchronization.
  • Scales well in distributed environments.
• Disadvantages:
  • Unpredictable data states; applications must handle inconsistencies.
  • Not suitable for systems requiring immediate accuracy.
• Use Case: Social media "like" counters, where slight delays in reflecting totals are acceptable.

---

3. Eventual Consistency

• Definition: A specific form of weak consistency where, given enough time and no new updates, all nodes will eventually reflect the same data. It promises convergence rather than immediate agreement.
• How It Works: Writes propagate asynchronously across nodes. Conflicts may arise but are resolved over time (e.g., via last-write-wins, version vectors, or manual reconciliation).
• Examples:
  • NoSQL databases like Cassandra, DynamoDB, or Riak.
  • Distributed caches like Memcached (with eventual replication).
• Advantages:
  • High availability and partition tolerance (aligned with the CAP theorem’s “AP” systems).
  • Good performance for read-heavy or geographically distributed systems.
• Disadvantages:
  • Temporary inconsistencies can confuse users or applications.
  • Conflict resolution logic may be complex.
• Use Case: E-commerce product catalogs, where slight delays in stock updates are tolerable.

---

Comparison of Strong vs. Weak/Eventual Consistency

Aspect	Strong Consistency	Weak Consistency	Eventual Consistency
Read Guarantee	Latest write always visible	No guarantee of latest data	Latest data eventually visible
Latency	Higher (due to sync)	Lower (async operations)	Lower (async propagation)
Availability	Lower (blocks on failure)	Higher	Higher
Complexity	Simpler for apps, harder for system	Harder for apps, simpler for system	Moderate for both
CAP Theorem	Prioritizes C (Consistency)	Prioritizes A (Availability)	Prioritizes A and P (Partition Tolerance)

---

Other Consistency Models

To provide a broader context, here are additional consistency levels often encountered:

• Causal Consistency: Ensures that causally related operations (e.g., a write followed by a read) are seen in the correct order, but unrelated operations may appear out of sync. Used in systems like COPS or Bayou.
• Read-Your-Writes Consistency: Guarantees that a client sees their own previous writes in subsequent reads, even if other clients see stale data. Common in session-based systems.
• Monotonic Reads Consistency: Ensures that if a client reads a value, it won’t see an older value in later reads (data moves forward). Useful in distributed file systems.
• Bounded Staleness: A hybrid model where reads may lag behind writes by a defined time or version threshold (e.g., Google Spanner).

---

Real-World Context

• Strong Consistency: Used in Google’s Spanner (with TrueTime for global synchronization) or traditional RDBMS for critical operations.
• Eventual Consistency: Powers Amazon DynamoDB (tunable consistency) and Netflix’s Cassandra deployment for user data.
• Weak Consistency: Seen in early peer-to-peer systems or applications where immediate accuracy isn’t critical.

Consistency choice depends on application needs. For example, a chat app might use eventual consistency for message delivery but strong consistency for user authentication. The CAP theorem (Consistency, Availability, Partition Tolerance—pick two) often guides these decisions in distributed systems.

---

Comparison of Consistency Models

Consistency Level	Guarantees	Performance Impact	Use Case
Strong Consistency	Always latest data	High latency	Banking, financial transactions
Eventual Consistency	Data converges over time	Low latency, high availability	Social media, caching
Causal Consistency	Maintains causal order	Medium latency	Chat apps, collaborative editing
Read-Your-Writes	User sees their own writes	Medium latency	Cloud storage, user preferences
Monotonic Reads	No time-travel reads	Medium latency	DNS, user sessions

Golang: Http POST Request with JSON Body example

Go standard library comes with "net/http" package which has excellent support for HTTP Client and Server.  

In order to post JSON body during post request, we need to convert the data to byte array format and send it along with the request.
You can convert the JSON to a byte array using "encoding/json" package. Then use the NewBuffer method to pass this byte array to the post method.

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

//If the struct variable names does not match with json attributes 
//then you can define the json attributes actual name after json:attname as shown below. 
type User struct {
	Name string  	`json:"name"`
	Job string 	`json:"job"`
}

func main(){

	//Create user struct which need to post.
	user := User{
		Name: "Test User",
		Job: "Go lang Developer",
	}

	//Convert User to byte using Json.Marshal
	//Ignoring error to
	body, _ := json.Marshal(user)

	//Pass new buffer for request with URL to post.
	resp, err := http.Post("https://reqres.in/api/users", "application/json", bytes.NewBuffer(body) )

	// An error is returned if there were too many redirects
	// or if there was an HTTP protocol error
	if err != nil {
		panic(err)
	}
	//Need to close the response stream, once response is read.
	//Hence defer close. It will automatically take care of it.
	defer resp.Body.Close()

	//Check response code, if New user is created then read response.
	if resp.StatusCode == http.StatusCreated {
		body, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			//Failed to read response.
			panic(err)
		}

		//Convert bytes to String and print
		jsonStr := string(body)
		fmt.Println("Response: ", jsonStr)

	} else {
		//The status is not Created. print the error.
		fmt.Println("Get failed with error: ", resp.Status)
	}
}

Golang: Http Get Request example

Go standard library comes with "net/http" package which has excellent support for  HTTP Client and Server.  

In this example we will be using "http.Get" method to execute Http Get method. We will convert the response to String and print it.

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func main(){

	//Use get method to hit the rest API
	resp, err := http.Get("https://reqres.in/api/users/2")

	// An error is returned if there were too many redirects
	// or if there was an HTTP protocol error
	if err != nil {
		panic(err)
	}
	//Need to close the response stream, once response is read.
	//Hence defer close. It will automatically take care of it.
	defer resp.Body.Close()

	//Check response code, if ok then read response.
	if resp.StatusCode == http.StatusOK {
		body, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			//Failed to read response.
			panic(err)
		}

		//Convert bytes to String and print
		jsonStr := string(body)
		fmt.Println("Response: ", jsonStr)

	} else {
		//The status is not ok. print the error.
		fmt.Println("Get failed with error: ", resp.Status)
	}
}

Its simple and easy to implement.

Golang: Reading environment variables

Usually, environment variables are used to provide the configuration information for the program. You can set the environment outside of the program and you can assess these during execution. Go provides the simplest mechanism to access the environment variables. 

You can use 'OS' package to get or set the environment variables.

package main

import (
	"fmt"
	"os"
	"strings"
)

func main() {
	//Read single environment variable value
	val := os.Getenv("WORKING_DIR")
	//Check if the value is set.
	if val != "" {
		fmt.Println("Working Dir is: ", val )
	} else {
		fmt.Println("Working Dir is not set. Setting it now.")
		//You can set env variable using setEnv method.
		os.Setenv("WORKING_DIR", "/root/app_dir")
		fmt.Println("Working Dir is: ", os.Getenv("WORKING_DIR") )
	}

	//You can also iterate over all the environment variables.
	//Here environment variables are returned as string containing varname=value form
	for _, varNVal := range os.Environ() {
		keyVal := strings.SplitN(varNVal, "=", 2)
		fmt.Println("Variable name: " + keyVal[0] + "\tValue: " + keyVal[1] )
	}
}

Golang: Write data to file

Writing data to file is the most common use case while writing a program.  Here we are going to see multiple ways of writing data to file.  

Write using ioutil.WriteFile

You can directly write data to file using ioutil.WriteFile" method. It will take care of creating file if the file does not exists.

package main

import (
	"fmt"
	"io/ioutil"
)

func main() {
	writeStringToFile("data.txt", "Hello File!!!")
}

func writeStringToFile(filePath string, data string) {

	//Need to convert string to byte array.
	//WriteFile will create and write data to file, if file does not exists.
	err := ioutil.WriteFile(filePath, []byte(data), 0644)
	checkNLogError(err)
}

func checkNLogError(err error){
	if err != nil {
		fmt.Println(err)
                panic(err)
	}
}

Write using WriteString

You need to create a file first, using the same file pointer we can write string to file.

package main

import (
	"fmt"
	"os"
)

func main() {
	writeStringToFile("data.txt", "Hello File!!!")
}

func writeStringToFile(filePath string, data string) {

	//Create a file to write data.
	f, err := os.Create(filePath)

	//If there is error to write to file, exit.
	checkNLogError(err)

	//Once file is opened, it should be closed.
	//Defer will take care of it, even if any error.
	defer f.Close()

	//Write data to file.
	_, err2 := f.WriteString(data)

	//Check if there is any error during writing data to file.
	checkNLogError(err2)
}

func checkNLogError(err error){
	if err != nil {
		fmt.Println(err)
		panic(err)
	}
}

Golang: Read data from File ( part II)

In the last post, we have seen how we can read data from the file line by line.  

Sometimes if the file size small then you can read the whole file in one go.  In this example, we will be reading the whole file in one go. 

We will be using 'ReadFile' from 'ioutil' lib to read the file content. 

package main

import (
	"fmt"
	"io/ioutil"
)

func main() {
	readFile("file.path")
}

func readFile(filePath string) {
	//Try to read file contents
	data, err := ioutil.ReadFile(filePath)
	//Check if there is any error reading file contents.
	if err != nil {
		fmt.Println("Unable to read the file content. Error: ", err)
		return
	}
	//Convert the byte data read from file to string and print.
	fmt.Println(string(data))
}

Read also

• Goang: Read data line by line
  
• Golang: Check if file exists
  

Golang: Read data from file line by line

Reading data from files is a common use case. Most of the time you need to read data line by line.

There are multiple ways to read data from the file.  

The simplest way is to open a file and use a scanner to read data line by line. The sample code below does the same. 

package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	readLineByLine("file_path")
}

func readLineByLine(filePath string) {
	f, err := os.Open(filePath)       //Open file for reading
	if err != nil {
		fmt.Println("Unable to open the file. Error: ", err)
		return
	}
    defer f.Close()     //Close file pointer after reading file.
    
    line := bufio.NewScanner(f)    //Scanner to read line by line
	for line.Scan() {
		fmt.Println(line.Text())
	}
}

Other related posts: 

• How to check if file with given path exist
• Read data from file part II

Golang: How to know IP Address and Hostname

Sometimes you want to know the hostname and IP address of the machine you are running the program on.  We can get the hostname using the OS package.

For IP address we can use the Net package. You can read all IP addresses using InterfaceAddrs, you need to filter out the loopback address to get real IP. 

package main

import (
	"fmt"
	"net"
	"os"
)

func main() {
	//Reading the hostname using OS package.
	hostname, err := os.Hostname()
	if err != nil {
		fmt.Println("Hostname: ", hostname)
	}

	
	//Reading IP Address 
	addrs, err := net.InterfaceAddrs()
	if err == nil {
		for _, a := range addrs {
			if ipnet, ok := a.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
				if ipnet.IP.To4() != nil {
					fmt.Println(ipnet.IP.String())
				}
			}
		}
	}
}

Golang: Check if file exists in go lang

It is easy to check if the file with the given path exists or not in Go lang. You can use 'Stat' method from 'os'.  Stat returns a FileInfo describing the named file. So if there is no error then you can safely assume the file exists.    

import (
	"fmt"
	"os"
)

func main() {
	if _, err := os.Stat("file.path"); err == nil {
		fmt.Println("File with given path exists.")
	} else if os.IsExist(err) {
		fmt.Printf("Path info error here. Error: %s", err.Error())
	} else {
		fmt.Println("File does not exist.")
	}
}

Fix: Buffer overflow when available data size...

This is an unusual problem you might face while consuming data from Kafka broker over SSL. 

java.lang.IllegalStateException: Buffer overflow when available data size (16384) >= application buffer size (16384)
	at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:592)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:448)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:398)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1308)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1248)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
	at kafka.tools.ConsumerPerformance$.consume(ConsumerPerformance.scala:126)
	at kafka.tools.ConsumerPerformance$.main(ConsumerPerformance.scala:56)
	at kafka.tools.ConsumerPerformance.main(ConsumerPerformance.scala)

This could happen because of the Java Security provider sequence differences. In order to fix this you need to check what is the sequence of Java security provider on client and server. In order to confirm this check "jre/lib/security/java.security" file under Java installation directory. It contains the Java Security provider classes sequence to use. Make sure both client and server has the same sequence.

Fix: Python requests.get(url) times out but works in browser

If Python requests.get fails with a timeout and the same URL works in the browser then you will need to simulate the browser request as is while invoking requests.get.  

Usually, this will get resolved if you put the "user-agent" header, which tells the server which browser is requesting. 

session = requests.Session()

headers = {
           'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36',          
      }

response = session.request('GET', url, headers=headers, allow_redirects=False)

If this does not resolve the issue then launch the Developer Tools in Chrome and launch the same URL in chrome. Now on Developer Tools check the network tab which shows the requests made by the browser and their details. Copy all the request headers from there and put it into the headers map above and try again.
This time you should not face the issue.

Most Useful Linux/Unix Commands

Here is the list of commonly used commands on Unix/Linux systems.

1. Split File based on number of lines 

  If you want to split text file "abc.txt" containing 50 lines by each line use the below command

csplit abc.txt 1 {49}

2. Find out Top 10 Memory intensive processes 

  If you want to figure out which are the top 10 memory consuming processes use the command below

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

You can change mem with cpu for top CPU consuming processes.

3. Curl command to skip SSL Certificate check (HTTPS)

You can skip the SSL Certificate check by providing -k parameter to curl as below.

curl -k --location --request GET 'https://your.web.com'

This post will be forever updated as and when I find new useful commands.

Kafka Performance Benchmarking

Apache Kafka comes with the kafka-*-perf-test tool, which can be used to benchmark the performance of the Kafka cluster. This can be used as the first level of benchmarking tool before you run your production load tests. 

Kafka Producer Test

You can use kafka-producer-perf-test to measure the Kafka write performance.  The below command demonstrates running the Kafka producer test for 100K records.

bin/kafka-producer-perf-test.sh \ 
--num-records 100000 --topic test2 \
--record-size 10 --throughput -1 \
--producer.config producer.properties

Once you run the above command you will see the result like below

100000 records sent, 158730.158730 records/sec (1.51 MB/sec), 104.48 ms avg latency, 202.00 ms max latency, 117 ms 50th, 154 ms 95th, 158 ms 99th, 159 ms 99.9th.

The producer.properties should contain details like below

linger.ms=100
acks=1
batch.size=1000
buffer.memory=4294967296
request.timeout.ms=300000
bootstrap.servers=localhost:9092

The "--help" option provides more details on controlling different parameters for the test.

Kafka Consumer Test

Similar to the producer test you can run a consumer test as well to measure the read performance from Kafka. The below command demonstrates consumer test on broker localhost and topic test2. 

bin/kafka-consumer-perf-test.sh \
--broker-list localhost:9092 \
--consumer.config consumer.properties \
--topic test2 --print-metrics --threads 1 \
--num-fetch-threads 1 --messages 10000 \
--show-detailed-stats

Once you run the consumer perf test with print-metrics it will provide consuming stats as well like below.

time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-27 16:33:37:363, 0, 0.0041, 0.0006, 1, 0.1597, 1603816414867, -1603816408604, 0.0000, 0.0000
2020-10-27 16:33:42:384, 0, 842.1655, 167.7278, 211001, 42023.5013, 0, 5021, 167.7278, 42023.5013

You can control different parameters of the consumer test by providing the consumer.properties file as specified below

security.protocol=PLAINTEXT
max.partition.fetch.bytes=30485760
fetch.max.bytes=51971520
fetch.min.bytes=5242880
max.poll.records=500
fetch.max.wait.ms=50
enable.auto.commit=false
auto.offset.reset=earliest

Error: Zip64 archives are not supported by springframework

Springboot application may fail to start with error "Zip64 archives are not supported" with giving you stack trace as below.

  
Caused by: java.lang.IllegalStateException: Zip64 archives are not supported
    at org.springframework.boot.loader.jar.CentralDirectoryEndRecord.getNumberOfRecords(CentralDirectoryEndRecord.java:124)
    at org.springframework.boot.loader.jar.JarFileEntries.visitStart(JarFileEntries.java:91)
    at org.springframework.boot.loader.jar.CentralDirectoryParser.visitStart(CentralDirectoryParser.java:89)
    at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:57)
    at org.springframework.boot.loader.jar.JarFile.(JarFile.java:121)
    at org.springframework.boot.loader.jar.JarFile.(JarFile.java:109)
    at org.springframework.boot.loader.jar.JarFile.createJarFileFromFileEntry(JarFile.java:287)
    at org.springframework.boot.loader.jar.JarFile.createJarFileFromEntry(JarFile.java:262)
    at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:250)
    ... 6 more

The main reason is "Springboot loader does not support Zip64 format jar's. You can find excellent details about the error at here.
In order to resolve this error, you need to check which Jar is built with Zip64 format. Once you locate the file which is built with Zip64, try to reduce the no of files in that jar to less than 65535, so that it will get build without Zip64. Once you fix this, your application will start working.

Quickest way to launch camera app on Samsung Galaxy Note 9

Camera is the mostly used app on smart phones, and its frequently required app. Having quick shortcut to launch the camera is always helpful. 
Samsung Galaxy Note 9 offers the always on display mode, where it allows to enable on screen home button.  Always on display also allows to set camera as a app to launch when double tap on home button. 

To enable this go to Setting -> Lock Screen -> Always on Display. On always on display settings choose action for "Double tap Home Button". To use this shortcut you should also select "Home button and clock" under Contents to shown on the same screen.

Enable/Disable always on Display on Samsung Galaxy Note 9

If you like to save your battery more, you can turn off the always on display on your Samsung Galaxy Note 9. 

To do so, go to Settings -> Display. Now on display settings scroll down to the end, you will find the always  on  Display  setting.
Other option is go to Settings. Now using search bar search for "Always on".

Here enable/disable always on display by using the toggle at the top. There are many settings for always on display. One more important is to customize the Always on  display timing. You can choose the time period when you want the always on  display  should be enabled. This is also another great way to save battery and still using  this feature.

Using Connect-standalone in Kafka with Kerberos cluster

Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It can also be used in secured Kafka environment. In Kerberixed Kafka installation also you can use the Kafka Connect utilities. 
You just have to provide the Java security config and Kerberos config as parameters to connect utils. 
Below command shows how can you specify the security properties to the connect-standalone.sh.
The command assumes that you are in the Kafka installation directory.

bin/connect-standalone.sh connect_standalone.properties source.properties -Djava.security.auth.login.config=kafka-jaas.config -Djava.security.krb5.conf=krb5.conf

Here the kafka-jass.config should specify the file path which contains the KafkaClient properties like below.

  KafkaClient {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=true
 keyTab=""
 storeKey=true
 useTicketCache=false
 serviceName="kafka"
 principal="";
};

And the krb5.conf should detail about the KDC server property. Sample shown below.

  
        [libdefaults]
 renew_lifetime = 7d
 forwardable = true
 default_realm = example.com
 ticket_lifetime = 24h
 dns_lookup_realm = false
 dns_lookup_kdc = false
 default_ccache_name = /tmp/krb5cc_%{uid}
 #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

 [realms]
 example.com = {
 master_kdc = my.master.kdc.com
 admin_server = my.admin.com
 kdc = my.kdc.com
 }

If above command does not work of if Kafka connect is not able to use the kerberos config, then update the Kafka-run-class.sh file to pass these security parameters to exec command at the end "-Djava.security.auth.login.config=kafka-jaas.config -Djava.security.krb5.conf=krb5.conf"

How to avoid host key checking during ansible script execution

During execution of ansible playbook or ansible script on new remote host (which you have not connected via ssh so far) the script might fail with below error

 UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ----}

To avoid this you can set the environment variable (ANSIBLE_HOST_KEY_CHECKING) to disable host check before ssh during ansible script execution.

 export ANSIBLE_HOST_KEY_CHECKING=False

Now you can execute the ansible script.

HMaster not starting up, error "java.io.IOException: Filesystem closed"

HMaster fails to start with below error

util.FSHDFSUtils: attempt=0 on file=hdfs://krbhdfs/apps/hbase/data/MasterProcWALs/state-00000000000000000001.log after 1ms java.io.IOException: Filesystem closed.

There could be multiple causes of the error. 

If you are willing to loose the data, remove manually the logs under "apps/hbase/MasterProcWALs" and  "apps/hbase/WALs" from HDFS. After doing this start the HMaster service.