Spark
When working with change data capture data, containing sometimes just the updated cells for a given PK, it is not easy to efficiently obtain the latest value for the entry (row). Let's dive into an example. First though, we need a couple of definitions about change data capture
kafka
The first big step to work with Kafka is to put data in a topic, and so is the purpose of this post. Being JSON the most common way to intercomunicate, and having the schema with the data, we will explore how we build a producer in scala to start
kafka
In this second part of the Kafka in MapR series we are going to de-duplicate identical messages using HBase. > NOTE I am using MapR so not all configurations are the same in its Open Source counterparts (e.g. I don't need to add the ZooKeeper nodes) This
kafka
While Spark continues to thrive as the main big data processing framework for batch and streaming, alternatives emerging from the 1970s actor model and the reactive manifesto are gaining notoriety. Akka is widely known in the Scala community and on March 2016 Confluent released its library Kafka Streams. One of
scala
Similar to the previous post Reading Sequences of Scala classes in Play JSON [https://8vi.cat/sequences-of-classes-in-play-json/], I found some issues when trying to parse and write Timestamp using the JSON module from Play Framework. The objective I want to be able to write and read Timestamp format. I have
scala
One wise man said once: "Try to improve 1% every day so you will be an expert without you noticing". Following this advise, in my personal projects I try to use new libraries and this time the time has come for JSON with Play Framework. The objective of
Spark
Recently I have been discussing with some colleagues the advantages of type-safe languages, the main reason I ditched python for Scala. Scala provides, for me the best of both worlds: the compiler is smart enough to determine variable types, and in the most cases it even can infer return types
scala
Some weeks ago I co-organized a Scala BCN meetup where we implemented a Bank account class in pairs and TDD. Easy for any of you that are starting with Scala. Here you have the github link [https://github.com/martsec/bankingKata] to try it by yourself! Copy from the README.
Spark
Graphs have been around quite a while. In fact, the first paper in the history of graph theory was written by Euler himself in 1736! However, it wasn't until 142 years later when the term graph was introduced in a paper published in Nature. Despite those shiny names,
Spark
A pivot table is an old friend of Business Intelligence: it allows to summarize the data of another table, usually by performing aggregations (sum, mean, max, ...) on aggregated data. They are found in multiple reporting tools (i.e. Qlik, Tableau and Excel) and analytical RDBMS (i.e. Oracle) also implement