21 Lessons for the 21st Century by Yuval Noah Harari

Yuval Noah Harari creates a useful framework for confronting these fears. While his previous best sellers, “Sapiens” and “Homo Deus,” covered the past and future respectively, his new book is all about the present. 

Yuval first started with the failing global stories: communism, fascism and liberalism/capitalism. Capitalism has took over the modern world ever since fascism failed during World War II by promising a larger slice of the pie for everyone to solve difficult social and political conflicts. But it has no answer to ecological collapse and technology disruption since the essence of capitalism is the main cause of both issues. Yuval suggests modern world need to evolve a new social and political model to adapt to the world change. The book then unfolds by covering the following issues:

Work – the alerting question – will AI replace human? Even consumers could be AI (for example, Google engine algorithm consume the web pages). Working class might see a shift from exploit to irrelevance – communism comeback is hard if the mass lose their economic value (government could slow down automation) since there’s is no “working class" left. There’s a debate on to whether provide people with universal basic income (capitalist paradise) or universal basic services (communist paradise). But what is basic? Who defines that? Even then, people might feel more inequality and lack of social mobility when comparing to people on top with extra stuff.

Liberty – follow your heart only started recent centuries thing as liberalism dominates the world. Now, not long after that, it might transfer to following the algorithm.

Equality –

Community – Industry revolution breaks the community. Core family still exists but not the community that used to tie people close. FaceBook announced their idea to build the community offline to solve social polarization. Yuval points out to achieve that, they have to do better at priority social concerns over financial interests and really bridge online to offline community.

Nationalism – human have turned clans and tribes into single nation. Not all culture fits in this mainstream structure. Nation cannot suffer. It is a fictional concept that simplifies the world to better control people. People are asked to sacrifice for their countries without questioning the authority. But what exactly are you sacrificing for?

Immigration – debates whether people should have the freedom to immigrate and host country should accept them and treat them no difference. The paradox of this issue is that host country enjoys the cheap labor but treat immigrants as second-class citizens.

Humility –

Secularism – From an ethical perspective, monotheism was arguably one of the worst ideas in human history. What monotheism undoubtedly did was to make many people far more intolerant than before. The late Roman Empire was as diverse as Ashoka’s India, but when Christianity took over, the emperors adopted a very different approach to religion. Religion has no answers to any of life’s important questions, which is why there is no great following for a Christian version of agriculture, or a Muslim version of economics. “We don’t need to invoke God’s name in order to live a moral life. Secularism can provide us with all the values we need.”

Ignorance –

Justice –

Emotion – Human make life and death decisions based on emotions. Emotions are formed by million of neuron reactions that make decisions for us. Desires is one of the emotions. We don’t not free will to determine what desires we have – they’re merely biochemical mechanisms, even though we have free will to achieve those desires. Our feelings are all based on calculation for survival and reproduction.

Post-Truth – religious stories are fake news that are repeated 1000 times and became truth. We argue that platforms like FaceBook are manipulating news, showing only the ones that are to the viewer’s favor even that happens with other platforms as well such as TV. But real manipulation is to cover an entire massacre from the people like some authorities do.

Science Fiction – In Brave New World, the authority utilized biotech to make sure all people are content, optimized with joy so no there’s rebels. People are split into different classes before born to keep the society stable. So, what’s wrong with that view? Like John the savage said in the novel: “But I don’t want comfort. I want God, I want poetry, I want real danger, I want freedom, I want goodness. I want sin."

Education – don’t be controlled by the technology, know what you want in life before it knows better than you, and use technology to help you get it.

Meaning – life is not a story. There’s no story. It’s just way to simplify difficult choices and control that mass more easily, such as nationalism prompts people to do every thing for the country; and ancient people believe the story that everything they do is for the afterlife. “The Buddha taught that the three basic realities of the universe are that everything is constantly changing, nothing has any enduring essence, and nothing is completely satisfying.  Suffering emerges because people fail to appreciate this … The big question facing humans isn’t ‘what is the meaning of life?’ but rather, ‘how do we get out of suffering?’ … If you really know the truth about yourself and about the world, nothing can make you miserable.  But that is of course much easier said than done.”

Meditate – observe what’s real. Reality can suffer, fake cannot. Human desire is merely chemical reactions. “I have the free will to achieve my desires, but not the free will to control my desires." It’s obvious that Yuval adores Buddhism meditation and rise above egoistic obsession. In fact, it’s the only thing he mentioned useful and without contradicting opinion on it. He spent two hours per day and 30 days per year meditating and observe the reality. Through the observation, Yuval believes that Nation cannot suffer. Think about it, what is France? It cannot suffer, hence it is not a real entity and pure fictional. Lastly, we need to understand the mind more before AI figures us out.

廣告

Go Training Part II – Method, Interface, and Encapsulation

We talked about Go function in the last particle, now we’d compare function with method. Functions with a pointer argument must take a pointer; while methods with pointer receivers take either a value or a pointer as the receiver when they are called. A function which takes a receiver is usually called a method. A method is like a function, but attached to a type (called as receiver). The official guide states “A method is a function with a special receiver argument” (https://stackoverflow.com/questions/8263546/whats-the-difference-of-functions-and-methods-in-go)

Methods

A method is like a function, but attached to a type (called as receiver). Methods are functions that are declared with a receiver which binds the method to a type. Methods can be used to operate on values or pointers of that type.The official guide states “A method is a function with a special receiver argument”. The receiver appears in between the func keyword and the method name. Consider the following example:

// u user is the notify method receiver – the type that receives this type -> notify is now attached to user type (self in other languages)

func (u user) notify()

// changeEmail implements a method with a pointer receiver.

func (u *user) changeEmail(email string) {

u.email = email

// declared with both value and pointer receivers. mutate variable bill (both pointer and value semantics)
bill := user{“Bill", “bill@email.com"}
bill.changeEmail(“bill@hotmail.com“)
}

• Key: Golang is pass by value, means fresh copies of the parameters are passed to each function/method call. To pass them by reference you can use pointers.
  • T and *T is distinctive separate types but can still call changeEmail() on user and notify() on *user -> some help from the compiler (adjust by doing de-reference, but be careful, it could be the way of adjust that are not intended)
• code smell: don’t mix type T and *T
  • if method used to mutate the pointer receiver: user *T (pointer receiver)
  • if not, user T (value receiver)
  • think about the nature of your type and be consistent:
  • pointer receiver: take address of the element:
    • <OS pacakge – file type> func (f *File) Chdir() error { // unique file to get the reference back to my file, use pointer everywhere when you interact with this type
    • where to use: unique, never mutate this file
  • value receiver:
    • usage:
      • a basic data structure
        • <time pacakge – time type> don’t take address of a bool or int, time is a more complex int
        • func Now() Time
        • does mutate it’s receiver (deep copied) and return at the end
      • a reference type: methods and functions that reference to a slice, map or interface (already a reference, no need to do pointers which is extra referencing)
• Deep copying stays on the stack

Maps

Maps provide a data structure that allow for the storage and management of key/value pair data.

• syntax:
  • make(map[string]user) // [type of key]type of value

users[“Roy"] = user{“Rob", “Roy"}

• users := map[string]user{“Roy":     {“Rob", “Roy"}} // syntax sugar
• map are optimized to be set instead of array
• map is unordered list, if need order, use slice of list. Iterating over a map is always random.
• Go can’t take address of an element in the map, instead, it takes the element and put it back in the map using hash(key), thus address links to elements is not deterministic (implementation details of map -> hash buckets is imbalanced)
  • There is restriction on map which it is not comparable (== and != cannot be applied on maps)
    • Slice, map, and function values are not comparable. (Go lang Spec: Comparison Operatior)
    • You also cannot implement your own comparable (no method overloading to keep it simple) but you can sort it by keys with “sort.Strings(keys)”
    • There is no method on maps, so you need to verbosely write it out with for loops:
      • for key, value := range users { fmt.Println(key, value)

Interface

Interfaces provide a way to declare types that define only behavior. This behavior can be implemented by concrete types, such as struct or named types, via methods. When a concrete type implements the set of methods for an interface, values of the concrete type can be assigned to variables of the interface type. Then method calls against the interface value actually call into the equivalent method of the concrete value. Since any concrete type can implement any interface, method calls against an interface value are polymorphic in nature.

• It is the way to define abstraction/polymorphism in Go. Polymorphism means that you write a certain program and it behaves differently depending on the data that it operates on.
• Go is not OOP language like Java and C++: no inheritance but have incapsulation and polymorphism
• abstraction:
  • define interface:

type reader interface { // abstraction with method signature

read(b []byte) (int, error) // need to implement this method

}

• concrete method read implements the reader interface for a network connection:

func (pipe) read(b []byte) (int, error) {}

• concrete type read implements the reader interface for a file.

func (file) read(b []byte) (int, error) {}

• no keyword “implements" when defining the type, code that uses the type will do the check
  • func retrieve(r reader) error
  • len, err := r.read(data)
• when compiler calls method that has reader r, what is actually doing is:
  • data -> data json/instruction -> iTable (points to fileType/methodSet that implements read()=actual instructions of the method to execute the concrete implementations)
  • https://research.swtch.com/interfaces complex graph of interface storage (contains pointer to data and type)
  • pointer receiver that implements the interface is a pointer that points back to the u user to mutate the data
• method sets

T	*T
value receiver	V	V (dereference)
point receiver	X (not consistent)	V

• iTable corresponds to the interface type, not the dynamic type
• sometimes need to have the concrete type: /Users/jeching/go/src/github.com/ardanlabs/gotraining/topics/go/design/composition/assertions/example1/example1.go
  • switch case to determine which concrete type
• empty interface() has no methods:
  • so all types at least implements the empty interface
  • represent any type
  • pass “sth.” into the interface

Constant

Constants are a way to create a named identifier whose value can never change. They also provide an incredible amount of flexibility to the language. The way constants are implemented in Go is very unique.

• Constants are not variables.
• They exist only at compilation.
• Untyped constants can be implicitly converted where typed constants and variables can’t.
• Think of untyped constants as having a Kind, not a Type.
• read-only variable
• part of the compiler, not lived in memory -> compiler level copy & paste, no address to where it lives
• They have a parallel type system.
• type Duration int64 // Duration use the int64 as memory model, Duration is not sub type of int64

Embedding

Embedding types provides the final piece of sharing and reusing state and behavior between types. Through the use of inner type promotion, an inner type’s fields and methods can be directly accessed by references of the outer type.

• what embedding is not:
  • inheritence – identity: A is on B is a C (not exist in Go!)
  • composition – ownership: types that contains other types – A (appHandler) has a B (database connection)
  • embedded type:

type admin struct { // admin has a user

user  // Embedded Type within the outer type admin level string

}

• The inner type’s method is promoted. no need to do the chain thing, 往上找
• ex: bufio.ReadWriter

Encapsulation

Packages contain the basic unit of compiled code. They define a scope for the identifiers that are declared within them. Exporting is not the same as public and private semantics in other languages. But exporting is how we provide encapsulation in Go. Code in go is compiled into packages and then linked together. Identifiers are exported (or remain unexported) based on letter-case. We import packages to access exported identifiers.

• Go: on the package level, not type level
• either exported(can access outside of the package) or unexported
• not specified by keywords but with case: type that starts with Capital letter
• can only export package level scope; cannot export local variables within a function
• package = folder, import path:
  • $GOPATH/src/{package}
  • cannot do * to import all -> be explicit
  • conflicts: add name to one of the package for reference
• go get {new packages} // command line to get new packages

Go Training Part I – Type System

Intro

Golang is designed to be relatively fast for both human and computers (machine assembly). The most common use cases are developing cloud, network, and tooling services. There are three tenets that Go values the most:

• Integrity: clear surpasses clever and magic
• readability
• productivity

Here is the language origin.  In a Go program, packages have to be main and import other packages if used. Go is a compiled language (do “go build" and get executable binary code) but act like interpret language (ex: Python, can do “go run main.go" to build binary in temp folder, execute and finally delete the folder). Below is three important features of Go:

• Go is build for hardware:
  • file {binary code} > tell you the executable required hardware
  • GOOS=linux go build
  • take out VM middle layer, care about the hardware it’s running on
  • GOOS=windows go build
• Go binaries are self-contained
  • ls -lh > 2MB for hello world
  • all package imported are built inside the binaries
  • great to build tooling: k8s, docker
• Go is opinionated
  • $GOPATH=~/go: prescribed structure project
    • exist for the past 10 years
    • might go away in .12
  • only one right way to do thing
  • everything go into ~/go/src folder

Type system

Variables

Variables are at the heart of the language and provide the ability to read from and write to memory. In Go, access to memory is type safe. This means the compiler takes type seriously and will not allow us to use variables outside the scope of how they are declared.

• • get from types:
    • representation (boolean=true or false)
    • storage (cost of the type, nothing is free)
    • every type has it’s zero (default) value
    • (bad) too many ways to declare variables in Go -> suggested way:
      • var a int // cost/storage for this line:  32 or 64 bits all zeros, depend on your compile architecture (0)
      • var b string // 8 bits point adress & 8 bits for the length (“” -> nil, 0)
      • var c float64 // cost/storage for this line:  64 bits all zeros (0.0)
      • var d bool // cost/storage for this line: 8 bits all zeros (false)
      • aa := 10 // go infer the type
    • type conversion (value copied into different extended memory)
      • aaa := int32(10) // not same as type casting – extend 1 byte to 4 bytes, could interfere with other variables
  • variables exist as long as there’s reference to it
  • don’t use package level/global variables
  • not FP language but can pass functions around
  • objects live on stack or heap: Go escape analysis
  • variable scope: exist within {}
• declare your own type with keyword “type”:
  • struct is the most common created type: type made of other types as composite fields
  • padding: (source)
    • key: minimize the padding by sorting the fields largest to smallest
    • example: How memory works internally? When you have a struct like this one :

type myStruct struct {
myBool   bool // 1 byte
myFloat float64  // 8 bytes
myInt  int32   // 4 bytes
}

As you see a boolean takes 1 byte, a float64 8 bytes, and an int32 4 bytes.

But the memory allocates consecutive packet of 8 bytes. So instead of taking 1 + 8 + 4 = 13 bytes. This struct will takes : 24 bytes because in memory we will have :

So 8 + 8 + 8 = 24 bytes.

It’s possible to optimize, ordering the struct by bytes taken:

type myStructOptimized struct {
myFloat float64 // 8 bytes
myBool  bool // 1 byte
myInt   int32 // 4 bytes
}
This new struct ordered will take now 16 bytes, because in memory it will be allocated like that (https://play.golang.org/p/gmkrt6X7aM) :

8 + 8 = 16 bytes

As you can see, there are a padding to respect the data alignment, (https://en.wikipedia.org/wiki/Data_structure_alignment), So it means that in some case the data is not optimally word-aligned (depending on many things processor, cache memory, virtual memory) and it can sometimes slower the application

Pointers

Everything is pass by value in Go. Pointers provide a way to share data across function boundaries. Having the ability to share and reference data with a pointer provides flexibility. It also helps our programs minimize the amount of memory they need and can add some extra performance. 

• • Use pointers to share data.
  • Values in Go are always pass by value.
  • “Value of", what’s in the box. “Address of" ( **&** ), where is the box.
  • The (*) operator declares a pointer variable and the “Value that the pointer points to".
  • operator of pointers in Go:
    • & is address of where is some value
    • * is to get value pointer points to what is the value (or dereference the pointer from the address and get to the thing at the other side)
    • count := 10
    • if pass in count (primatives), Go will not modify the value of count. If pass in &count (address of count), Go will modify the actual count value
  • Go escape analysis:  
    • The Go compiler is smart enough to automatically decide whether a variable should be allocated on the heap, where it will later need to be garbage collected, or whether it can be allocated as part of the stack frame of the function which declared it. Stack-allocated variables, unlike heap-allocated variables, don’t incur any GC overhead because they’re destroyed when the rest of the stack frame is destroyed – when the function returns.
    • Go’s escape analysis is more basic than the HotSpot JVM, for example. The basic rule is that if a reference to a variable is returned from the function where it is declared, it “escapes” – it can be referenced after the function returns, so it must be heap-allocated. This is complicated by:
      • functions calling other functions
      • references being assigned to struct members
      • slices and maps
      • cgo taking pointers to variables
  • example:

// createUserV1 creates a user value and passed a copy back to the caller.

func createUserV1() user {

u := user{

name:  “Bill",

email: “bill@ardanlabs.com",

}

return u

// createUserV2 creates a user value and shares the value with the caller.

func createUserV2() *user {

u := user{

name:  “Bill",

email: “bill@ardanlabs.com",

}

return &u // &u escapes to heap： moved to heap

• stack vs heap
  • Go deal with memory collection for you
    • stack starts as 2k
    • How Stacks are Handled in Go (source)
    • Goroutines are cheap, cooperatively scheduled threads of execution that are used for a variety of operations, like timeouts, generators and racing multiple backends against each other. To make goroutines suitable for as many tasks as possible, we have to make sure that each goroutine takes up the least amount of memory, but also have people be able to start them up with a minimal amount of configuration.
    • Instead of giving each goroutine a fixed amount of stack memory, the Go runtime attempts to give goroutines the stack space they need on demand, freeing the programmer from having to make decisions about stack size

Array

Arrays are a special data structure in Go that allow us to allocate contiguous blocks of fixed size memory. Arrays have some special features in Go related to how they are declared and viewed as types.

• Arrays are fixed length data structures that can’t change.
• Arrays of different sizes are considered to be of different types.

• • make an array:
    • friends := [5]string{“Annie", “Betty", “Charley", “Doug", “Edward"}
  • iterate the array:
    • for i, v := range friends { // do stuff with i & v}
  • array len is built-in GO
  • L1 cache is fastest

L1 cache reference                           0.5 ns

Branch mispredict                            5 ns

L2 cache reference                           7 ns 14x L1 cache

Mutex lock/unlock                           25 ns

Main memory reference                      100 ns 20x L2 cache, 200x L1 cache

Compress 1K bytes with Zippy             3,000 ns 3 us

Send 1K bytes over 1 Gbps network       10,000 ns 10 us

Read 4K randomly from SSD*             150,000 ns    150 us ~1GB/sec SSD

Read 1 MB sequentially from memory     250,000   ns 250 us

Round trip within same datacenter      500,000 ns 500 us

Read 1 MB sequentially from SSD*     1,000,000   ns 1,000 us    1 ms ~1GB/sec SSD, 4X memory

Disk seek                           10,000,000 ns 10,000 us 10 ms 20x datacenter roundtrip

Read 1 MB sequentially from disk    20,000,000   ns 20,000 us   20 ms 80x memory, 20X SSD

Send packet CA->Netherlands->CA    150,000,000 ns 150,000 us 150 ms

Slice

Slices are an incredibly important data structure in Go. They form the basis for how we manage and manipulate data in a flexible, performant and dynamic way. It is made useful for the programmers.

• • Slices are like dynamic arrays with special and built-in functionality.
  • There is a difference between a slices length and capacity and they each service a purpose.
  • Slices allow for multiple “views" of the same underlying array.
  • Slices can grow through the use of the built-in function append.
  • make a slice: Create a slice with a length of 5 elements and a capacity of 8.
    • fruits := make([]string, 5, 8)
  • slice of slice:
    • slice := []string{“Apple", “Orange", “Banana", “Grape", “Plum"}
    • takeOne := slice[2:3]
  • built in function append will create a new array copied from the original slIce and update it’s reference to point to new array, L=C=1 (capacity update 1->2->4->8->16…-> table doubling: guarantee continuous memory
  • allow programmer to do re-slicing

Functions

Functions are at the core of the language. They provide a mechanism to group and organize our code to separate and distinct pieces of functionality. They can be used to provide an API to the packages we write and are a core component to concurrency.

• • Functions can return multiple values and most return an error value.
  • The error value should always be checked as part of the programming logic.
  • The blank identifier can be used to ignore return values.
  • usual func type:
    • func()
    • func() return val
    • func() return error
    • func getUser(name string) (string, error)
      • functions can return multiple values (error – built in type)
      • functions can accept arg but not using to satisfy interface

// Make a call to get the user in a json response.

r, err := getUser(name)

if err != nil { // if failed, return err and pass it up the stack

return nil, err

}

• • Defer func
    • A defer statement defers the execution of a function until the surrounding function returns.
    • The deferred call’s arguments are evaluated immediately, but the function call is not executed until the surrounding function returns.
    • https://tour.golang.org/flowcontrol/12
    • ex:

// Declare an anonymous function and assign it to a variable.

f := func() {

fmt.Println(“Variable:", n)

}

// Call the anonymous function through the variable.

f()

// Defer the call to the anonymous function till after main returns. -> like callbacks in JS, order is FILO

defer func() {

fmt.Println(“Defer 1:", n)

}()

• • func json.Unmarshal (data []byte, v interface{}) error
    • Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v. If v is nil or not a pointer, Unmarshal returns an InvalidUnmarshalError.
    • To unmarshal JSON into a struct, Unmarshal matches incoming object keys to the keys used by Marshal (either the struct field name or its tag), preferring an exact match but also accepting a case-insensitive match
    • https://golang.org/pkg/encoding/json/#Unmarshal
    • ex:

// retrieveUser retrieves the user document for the specified user and returns a pointer to a user type value.

func retrieveUser(name string) (*user, error) {

// Make a call to get the user in a json response.

r, err := getUser(name)

if err != nil { // if failed, return sth with that err and pass it up the stack

return nil, err

}

// Unmarshal the json document into a value of the user struct type.

var u user

err = json.Unmarshal([]byte(r), &u)

return &u, err

}

// GetUser simulates a web call that returns a json document for the specified user.

func getUser(name string) (string, error) { // function can return multiple values (error – built in type)

// accept arg but not using it to satisfy the interface

response := `{“id":1432, “name":"sally"}`

return response, nil

}

Java Generics

Have you ever wrote Java code like List<Long> list = new ArrayList<>(); ? Then you might already be using generics without realizing. It is not too difficult to make use of the generic types and methods provided by the JDK. Writing your own generic types could be a bit more difficult. In this article, we will cover some basics of Java Generics and example of refactoring our code with generics to illustrate why we need them.

Java Generics Basics

Generics are implemented using Type Erasure

In Java a class or an interface can be declared to define one or more type parameters and those type parameters should be provided at the time of object construction. For example,
List<Long> list = new ArrayList<>();
list.add(Long.valueOf(1));
list.add(Long.valueOf(2));
In the example shown above a List is created which can only contain elements of type Long and if you try to add any other type of element to this list, it will give you compile time error. It helps detect errors at compile time and makes your code safe. Once this piece of code is compiled ,the type information is erased resulting into similar byte code as we would have got if the same piece of code was written using Java 1.4 and below. This results in binary compatibility between different versions of Java. So, a List or List<> are all represented at run-time by the same type, List.

Generics are implemented by erasure. This means that they enforce their type constraints only at compile time and discard (or erase) their element type information at runtime. Arrays are covariant and reified; generics are invariant and erased. As a consequence, arrays provide runtime type safety but not compile-time type safety, and vice versa for generics. As a rule, arrays and generics don’t mix well. If you find yourself mixing them and getting compile-time errors or warnings, your first impulse should be to replace the arrays with lists

Use of wildcards with extends or super to increase API flexibility

There are times that you need to work not only with T but also with sub types of T. For example, the addAll method in the Collection interface which adds all the elements in the specified collection to the collection on which it is called. addAll method has the signature
boolean addAll(Collection <? extends E> c)
This ? extends E makes sure that you can not only add collection of type E but also of subtype of E. ? is called the wildcard and ? is said to be bounded by E. So,if we create a List of Number then we can not only add List of number but we can also add List of Integer or any other subtype of Number.
List numbers = new ArrayList();ArrayList integers = new ArrayList();
ArrayList longs = new ArrayList();
ArrayList floats = new ArrayList();
numbers.addAll(integers);
numbers.addAll(longs);
numbers.addAll(floats);
So far we have covered the use of extends with wildcards and we saw that API became more flexible after using extends . But where will we use super? Collections class has a method called addAll which add all the specified elements to the specified collection. It has following signature
public static <T> boolean addAll(Collection<? super T> c, T… elements) ;
In this method you are adding elements of type T to the collection c. super is used instead of extends because elements are added into the collection c whereas in the previous example of Collection interface addAll method elements were read from the collection . In the Effective Java book, Joshua Bloch calls this PECS . PECS stands for producer (read elements from the collection) extends, consumer super (add elements to the collection).

Use of Multiple Bounds

Multiple bounds is one of the generics features which most developer do not know. It allows a type variable or wildcard to have multiple bounds. For example, if you to define constraint such as that the type should be a Number and it should implement Comparable.
public static <T extends Number & Comparable<? super T>> int compareNumbers(T t1, T t2){return t1.compareTo(t2);}

It makes sure that you can only compare two numbers which implement Comparable. Multiple bounds follows the same constraints as followed by the a class i.e. T can’t extend two classes ,you have to first specify the class then the interface, and T can extend any number of interfaces.
public static <T extends String & Number > int compareNumbers(T t1, T t2) // does not work..can’t have two classes
public static <T extends Comparable<? super T> & Number > int compareNumbers(T t1, T t2) // does not work..
public static <T extends CharSequence & Comparable<T>> int compareNumbers(T t1, T t2)// works..multiple interfaces
public static <T extends CharSequence & Comparable<T>> int compareNumbers(T t1, T t2)// works..multiple interfaces

Sources

http://www.angelikalanger.com/GenericsFAQ/JavaGenericsFAQ.html
https://dzone.com/articles/5-things-you-should-know-about-java-generics
Effective Java, 3rd Edition

Use Kafka Streams With Java Reactive Streams

https://danlebrero.com/2017/01/05/proof-of-concept-using-kafkastreams-and-ktables/

In the last article, we used kafka consumer and producer to read in and write out the kafka messages. In this article, we will show how to replace them with Kafka Streams so that we don’t have to handle back pressure ourselves. Back-pressure is an important feedback mechanism that allows systems to gracefully respond to load rather than collapse under it.

Kafka Streams is a new open source library, part of the Apache Kafka project, that promises to make stream processing simple without losing the power and scalability of other stream processing systems like Storm or Spark Streaming

The major selling points are:

• Scalable, using the same partition-based model as Kafka.
• Real time, optional micro-batching.
• Stateful stream processing.
• It is a library, not a framework.

Kafka Streams don’t need any new infrastructure, depending only on the Kafka cluster. It has a nice functional API similar to Java 8 reactive streams.

Configure Kafka Streams

To start Kafka streams in your Java project, you’d need to define the following properties:

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, kafkaStreamConfig.getApplicationID());
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaStreamConfig.getBootstrapServer());
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

StreamsBuilder builder = new StreamsBuilder();
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

You’d also need to specify the POJO deserializer forKafka stream when reading in the messages as dataPointSerde:

Map<String, Object> serdeProps = new HashMap<>();

final Serializer<DataPoint> dataPointSerializer = new JsonPOJOSerializer<>();
serdeProps.put("JsonPOJOClass", DataPoint.class);
dataPointSerializer.configure(serdeProps, false);

final Deserializer<DataPoint> dataPointDeserializer = new JsonPOJODeserializer<>();
serdeProps.put("JsonPOJOClass", DataPoint.class);
dataPointDeserializer.configure(serdeProps, false);

final Serde<DataPoint> dataPointSerde = Serdes.serdeFrom(dataPointSerializer, dataPointDeserializer);

Introducing Java Reactive Streams

http://www.reactive-streams.org/

Java Reactive Streams is useful for handling streams of data—especially “live” data whose volume is not predetermined—requires special care in an asynchronous system. The most prominent issue is that resource consumption needs to be controlled such that a fast data source does not overwhelm the stream destination. Asynchrony is needed in order to enable the parallel use of computing resources, on collaborating network hosts or multiple CPU cores within a single machine (see more on Reactive Manifesto). The scope of Reactive Streams is to find a minimal set of interfaces, methods and protocols that will describe the necessary operations and entities to achieve the goal—asynchronous streams of data with non-blocking back pressure.

First, to transform the input Kafka Stream message, we can the KStream interface ValueMapper and define the logic for transform input DataPoint to LabeledDataPoint in the apply method.

public class LabeledDataPointProducer implements ValueMapper<DataPoint, LabeledDataPoint> {

    private final ShotoClassifier classifier;
    private final DataPointCassandraDao cassandraDao;

    public LabeledDataPointProducer(ShotoClassifier classifier, DataPointCassandraDao cassandraDao) {
        this.classifier = classifier;
        this.cassandraDao = cassandraDao;
    }

    /**
     * Obtains a label for the datapoint provided either through cache or classification
     * @param dataPoint
     * @return a LabeledDataPoint
     */
    public LabeledDataPoint apply(DataPoint dataPoint) {
        LabeledDataPoint cached = cassandraDao.getCached(dataPoint.getMd5String());
        // if cached then set isCached to true and return the labeledDataPoint
        if (cached != null) {
            cached.setCached(true);
            return cached;
        }

        // if not cached then transform and save the labeledDataPoint to cassandra database
        LabeledScore<CategoryClassification> label = classifier.classify(dataPoint.getInputText()).getLabel();
        LabeledDataPoint ldp = LabeledDataPoint.fromClassification(dataPoint, label);
        cassandraDao.saveDataPoint(ldp, ldp.isHuman());
        return ldp;
    }
}

We can implement Consumer interface (consume input, return void) that takes the type of kafka message “LabeledDataPoint" and pass into the checking logic defined in the override accept method:

public class OutboundConsumer implements Consumer<LabeledDataPoint> {

    final Random random = new Random(1L);
    private double confidenceThreshold;
    private double highConfAuditPct;
    private String highConfTag;
    private String lowConfTag;
    private BiConsumer<LabeledDataPoint, String> writeToOutbound;

    public OutboundConsumer (double confidenceThreshold, double highConfAuditPct, String highConfTag, String lowConfTag, BiConsumer<LabeledDataPoint, String>writeToOutbound) {
        this.confidenceThreshold = confidenceThreshold;
        this.highConfAuditPct = highConfAuditPct;
        this.highConfTag = highConfTag;
        this.lowConfTag = lowConfTag;
        this.writeToOutbound =writeToOutbound;
    };

    @Override
    public void accept(LabeledDataPoint labeledDataPoint) {
        if (labeledDataPoint.isCached()) {
            return;
        } else if (labeledDataPoint.getLabelConfidence() < confidenceThreshold) {
            writeToOutBound.accept(labeledDataPoint, lowConfTag);
        } else if (random.nextFloat() < highConfAuditPct) {writeToOutBound.accept(labeledDataPoint, highConfTag);
        }
    }

    @Override
    public Consumer<LabeledDataPoint> andThen(Consumer<? super LabeledDataPoint> after) {
        return null;
    }
}

After preparing all the above, we can now pass in Java Reactive Streams to in Kafka Stream API to transform our input data as we intended with:

// read in and transform input DataPoint to LabeledDataPoint with ValueMapper
KStream<String, LabeledDataPoint> dataPointKStream = builder.stream(kafkaStreamConfig.getStreamInboundTopic(),
        Consumed.with(Serdes.String(), dataPointSerde)).mapValues(labeledDataPointProducer);

// check condition and write result to kafka streams
dataPointKStream.filter((k,v) -> v.isCached()).filter((k,v) -> v.getLabelConfidence() > 0.1)
        .to(kafkaStreamConfig.getStreamOutboundTopic());

// pass in Java consumer 
dataPointKStream.foreach((k,v) -> outboundConsumer.accept(v));

Spin Up Kafka Cluster With Kafka Connect

https://www.sohamkamani.com/blog/2017/11/22/how-to-install-and-run-kafka/

download version kafka_2.11-2.0.0 from here: 

Starting Zookeeper

Zookeeper is a key value store used to maintain server state. Kafka requires a Zookeeper server in order to run, so the first thing we need to do is start a Zookeeper instance.

Start the server by running:

bin/zookeeper-server-start.sh config/zookeeper.properties

Starting our brokers

Kafka brokers form the heart of the system, and act as the pipelines where our data is stored and distributed.

There are 3 properties, that have to be unique for each broker instance:

broker.id=0
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs

Since we will have 3 servers, it’s better to maintain 3 configuration files for each server. Copy the config/server.properties file and make 3 files for each server instance:

cp config/server.properties config/server.1.properties
cp config/server.properties config/server.2.properties
cp config/server.properties config/server.3.properties

Change the above 3 properties for each copy of the file so that they are all unique.

server.1.properties

broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs1

server.2.properties

broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs2

server.3.properties

broker.id=3
listeners=PLAINTEXT://:9095
log.dirs=/tmp/kafka-logs3

Also, create the log directories that we configured:

mkdir /tmp/kafka-logs1
mkdir /tmp/kafka-logs2
mkdir /tmp/kafka-logs3

Finally, we can start the broker instances. Run the below three commands on different terminal sessions: (start kafka process, different from kafka connect)

bin/kafka-server-start.sh config/server.1.properties
bin/kafka-server-start.sh config/server.2.properties
bin/kafka-server-start.sh config/server.3.properties

Creating a topic

Before we can start putting data into your cluster, we need to create a topic to which the data will belong with command:

bin/kafka-topics.sh –create –topic my-kafka-topic –zookeeper localhost:2181 –partitions 3 –replication-factor 2

The paritions options lets you decide how many brokers you want your data to be split between. Since we set up 3 brokers, we can set this option to 3.

The replication-factor describes how many copies of you data you want (in case one of the brokers goes down, you still have your data on the others).

The producer instance

The “producer” is the process that puts data into our Kafka cluster. The command line tools in the bin directory provide us with a console producer, that inputs data into the cluster every time your enter text into the console.

To start the console producer, run the command:

bin/kafka-console-producer.sh –broker-list localhost:9093,localhost:9094,localhost:9095 –topic my-kafka-topic

The broker-list option points the producer to the addresses of the brokers that we just provisioned, and the topic option specifies the topic you want the data to come under.

You should now see a command prompt, in which you can enter a bunch of text which gets inserted into the Kafka cluster you just created every time you hit enter.

Consumers

The only thing left to do is read data from the cluster with the command:

bin/kafka-console-consumer.sh –bootstrap-server localhost:9093 –topic my-kafka-topic –from-beginning

The bootstrap-server can be any one of the brokers in the cluster, and the topic should be the same as the topic under which you producers inserted data into the cluster.

The from-beginning option tells the cluster that you want all the messages that it currently has with it, even messages that we put into it previously.

When you run the above command, you should immediately see all the messages that you input using the producer, logged onto your console.

Kafka Connect

Kafka Connect, an open source component of Apache Kafka, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.

Using Kafka Connect you can use existing connector implementations for common data sources and sinks to move data into and out of Kafka.

Source Connector:

A source connector ingests entire databases and streams table updates to Kafka topics. It can also collect metrics from all of your application servers into Kafka topics, making the data available for stream processing with low latency.

Sink Connector:

A sink connector delivers data from Kafka topics into secondary indexes such as Elasticsearch or batch systems such as Hadoop for offline analysis.

You’d need to run Kafka connect in a separate thread with command:

./bin/connect-distributed.sh config/connect-distributed.properties

Then create a kafka topic with REST API:

$ curl -X POST -H “Content-Type: application/json" -d “$(cat test_sourceConnector.json)" localhost:8083/connectors

A sample connector config would look like this:

{
“name": “my-test-source",
“config": {
“connector.class": “io.confluent.connect.jdbc.JdbcSourceConnector",
“tasks.max": “1″,
“connection.url": “jdbc:sqlserver://{connect url};database={myDB}",
“connection.user": “username",
“connection.password": “password",
“mode": “incrementing",
“incrementing.column.name": “id",
“key.converter": “org.apache.kafka.connect.json.JsonConverter",
“topic.prefix": “my-test-jdbc-“,
“name": “my-test-source",
“table.whitelist": “myTable" // target table to retrieve data from
}
}

Principals Life and Work by Ray Dalio

The book <Principals Life and Work> is wrote by the founder of Bridgewater, Ray Dalio. It describes how Ray set up his fundamental life principles and how those got applied to management. I benefit from his perspective as I try to set my own goals for the next quarter both for work and more about life. The following is some quotes from the book that I would like to share:

While most others seem to believe that learning what we are taught is the path to success, I believe that figuring out for yourself what you want and how to get it is a better path.

While most others seem to believe that having answers is better than having questions, I believe that having questions is better than having answers because it leads to more learning.

While most others seem to believe that mistakes are bad things, I believe mistakes are good things because I believe that most learning comes via making mistakes and reflecting on them.

While most others seem to believe that finding out about one’s weaknesses is a bad thing, I believe that it is a good thing because it is the first step toward finding out what to do about them and not letting them stand in your way.

While most others seem to believe that pain is bad, I believe that pain is required to become stronger.

I believe that our society’s “mistakephobia” is crippling, a problem that begins in most elementary schools, where we learn to learn what we are taught rather than to form our own goals and to figure out how to achieve them. We are fed with facts and tested and those who make the fewest mistakes are considered to be the smart ones, so we learn that it is embarrassing to not know and to make mistakes. Our education system spends virtually no time on how to learn from mistakes, yet this is critical to real learning. As a result, school typically doesn’t prepare young people for real life—unless their lives are spent following instructions and pleasing others. In my opinion, that’s why so many students who succeed in school fail in life.

What I wanted was to have an interesting, diverse life filled with lots of learning—and especially meaningful work and meaningful relationships. I feel that I have gotten these in abundance and I am happy by 1) working for what I wanted, not for what others wanted me to do; 2) coming up with the best independent opinions I could muster to move toward my goals; 3) stress-testing my opinions by having the smartest people I could find challenge them so I could find out where I was wrong; 4) being wary about overconfidence, and good at not knowing; and 5) wrestling with reality, experiencing the results of my decisions, and reflecting on what I did to produce them so that I could improve. I believe that by following this approach I moved faster to my goals by learning a lot more than if I hadn’t followed it.

I have become someone who believes that we need to deeply understand, accept, and work with reality in order to get what we want out of life. Whether it is knowing how people really think and behave when dealing with them, or how things really work on a material level—so that if we do X then Y will happen—understanding reality gives us the power to get what we want out of life, or at least to dramatically improve our odds of success. In other words, I have become a “hyperrealist.”

Enjoying your job, a craft, or your favorite sport comes from the innate satisfaction of getting better. the sequence of 1) seeking new things (goals); 2) working and learning in the process of pursuing these goals; 3) obtaining these goals; and 4) then doing this over and over again is the personal evolutionary process that fulfills most of us and moves society forward.

<Making Choices>

Life consists of an enormous number of choices that come at us and that each decision we make has consequences, so the quality of our lives depends on the quality of the decisions we make.

The way we make our dreams into reality is by constantly engaging with reality in pursuit of our dreams and by using these encounters to learn more about reality itself and how to interact with it in order to get what we want.

For most people success is evolving as effectively as possible, i.e., learning about oneself and one’s environment and then changing to improve. Personal evolution is both the greatest accomplishment and the greatest reward.

There are five big types of choices that we continually must make that radically affect the quality of our lives and the rates at which we move toward what we want. People who choose what they really want, and avoid the temptations and get over the pains that drive them away from what they really want, are much more likely to have successful lives.

Those who react well to pain that stands in the way of getting to their goals—those who understand what is causing it and how to deal with it so that it can be disposed of as a barrier—gain strength and satisfaction. This is because most learning comes from making mistakes, reflecting on the causes of the mistakes, and learning what to do differently in the future.

As you design and implement your plan to achieve your goals, you may find it helpful to consider that:

• Life is like a game where you seek to overcome the obstacles that stand in the way of achieving your goals.

• You get better at this game through practice.

• The game consists of a series of choices that have consequences.

• You can’t stop the problems and choices from coming at you, so it’s better to learn how to deal with them.

• You have the freedom to make whatever choices you want, though it’s best to be mindful of their consequences

• The pain of problems is a call to find solutions rather than a reason for unhappiness and inaction, so it’s silly, pointless, and harmful to be upset at the problems and choices that come at you (though it’s understandable).

• We all evolve at different paces, and it’s up to you to decide the pace at which you want to evolve.

• The process goes better if you are as accurate as possible in all respects, including assessing your strengths and weaknesses and adapting to them.

 

The best advice I can give you is to ask yourself what do you want, then ask ‘what is true’—and then ask yourself ‘what should be done about it.’ I believe that if you do this you will move much faster toward what you want to get out of life than if you don’t!