Spark

Easy way to accessing s3 buckets with spark

It's sometimes difficult to access s3 files in Apache Spark if you don't use a prebuilt environment like Zepelin, Glue Notebooks, HUE, Databriks Notebooks or other alternatives. And googling around might get you half working solutions. But do not worry, I'll show you how easy it is to do it

Spark

Loading packages in pyspark jupyter notebook/lab

Using spark in an interactive way it's a bit cumbersome sometimes if you don't want to go to the good old terminal and you decide something like a jupyter notebook better suits you. Or you are doing a more complex analysis also. If that is the case and you don't

ai

Enable GPU access to WSL2 debian for your AI workloads

This post aims to centralize the information on how to use your Nvidia GPUs on debian using WSL2 in order to train and run your AI/ML models.

Spark

Grid neighbour operations in Spark

In physics or biology you sometimes simulate processes in a 2 dimensional lattice, or discrete space. In those cases you usually compute some local interactions of "cells", and with that, calculate a result. An example of this could be the Ising model which was proposed in 1920 for ferromagnetism in

travel

Canary Islands: Fuerteventura

Some pictures taken in Fuerteventura, Canary Islands. Use them as you  please with only one restriction: if used in public (blog, presentation  , news article...), cite me or this web/post. The consequences of uncontrolled planning and an economic crisis.That is NOT sandThis is actually calcified coral. "Palomitas"Why there

privacy

Com esborrar casa teva de Google Maps

Fa uns quants anys haviem de mirar mapes o fins i tot preguntar a desconeguts (on s'ha vist!) per poder arribar a llocs nous o que no recordàvem on eren, i ara només hem de buscar el nom del lloc i automàticament ens diuen on és i quina és la

travel

Canary Islands: Gran Canaria

Some pictures taken in Gran Canaria, Canary Islands. Use them as you  please with only one restriction: if used in public (blog, presentation  , news article...), cite me or this web/post. Dunas de MaspalomasThat is an odd museum, its purpose is to show the "scientific" studies done on native canary

ai

An overview on the EU Proposal for an Artificial Intelligence Act

Recently the European Commission released their proposal for a regulation laying down harmonized rules on Artificial Intelligence. After a review, I've tried to summarize its contents and explain what is considered an AI system, its classification, measures and finally give my opinion on the proposal. Fist we need to start

travel

Canary Islands: La Gomera

Some pictures taken in La Gomera, Canary Islands. Use them as you  please with only one restriction: if used in public (blog, presentation  , news article...), cite me or this web/post. El Teide is always present

travel

Canary Islands: Tenerife

Some pictures taken in Tenerife, Canary Islands. Use them as you  please with only one restriction: if used in public (blog, presentation  , news article...), cite me or this web/post. Wildlife

python

3 days deep dive with building websites and API in python and flask

For a personal project I've recently spend 3 days learning and coding a website+API with flask. Those have been some of the more enjoyable days I've had in a while (referring to coding). In this post, I explain you my takeaways on how to build better architected apps with

wsl

WSL handy commands

Often WSL2 does not work well anymore, and those commands have saved me some time and headaches. If you have messed-up something (e.g. sudo), login to the WSL as root (from windows CMD) C:\>wsl -u root Restart WSL2, sometimes needed when you can't connect from your windows to

datascience

Classifying architectural elements using AI (ML and neural networks): problem statement

I've recently finished the IBM's Avanced data science specialization [https://www.coursera.org/specializations/advanced-data-science-ibm] and as capstone project I thought about being able identify the architectural elements present in an image: is it a column? A vault? Or a bell tower? Let's see what can we achieve! The problem

tensorflow

Tensorflow model fails to train with initialize error

This post will provide some troubleshooting resources if you find yourself with this UnknownError when training an algorithm: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. Environment: * Windows 10 * Nvidia RTX GPU

Spark

Set-up pyspark in Mac OS X and Visual Studio Code

After reading this, you will  be able to execute python files and jupyter notebooks that execute Apache Spark code in your local environment. This tutorial applies to OS X and Linux systems. We assume you already have knowledge on python and a console environment. 1. Download Apache Spark We will

opinion Featured

The Theology of Artificial Intelligence

Artificial Intelligence systems are every day more present in our lives. Despite interacting continuously with them we are usually not conscious about how they work or what are the consequences of using this. In our mind, it's just a black box.

Spark

Compute 1-d histogram in spark

Spark has a way to compute the histogram, however, it is kept under low level and sometimes obscure classes. In this post I'll give you a function that provides you with the desired values passing a dataframe. In the official documentation the only mention [https://spark.apache.org/docs/latest/

Spark

Remember to execute the windowing once with Spark

This is one of the things it makes sense when you stop to think about it. When you are performing the same aggregation over a window in more than one column, it is recommended to execute only once that windowing. Lets dive in with an example: I am working with

Spark

Configure Nginx as reverse proxy for SparkUi

Nginx is a common webserver to be used as reverse proxy for things like adding TLS, basic authentication and forwarding the requests to other internal servers on your network. In this case, we are going to serve the sparkUI adding security (https) and authentication, and serving it in a different

Spark

Obtaining latest PK value Windows vs Joins with spark

When working with change data capture data, containing sometimes just the updated cells for a given PK, it is not easy to efficiently obtain the latest value for the entry (row). Let's dive into an example. First though, we need a couple of definitions about change data capture (CDC): > Change

travel

Madagascar: Animals

Some pictures taken in Madagascar. Use them as you  please with only one restriction: if used in public (blog, presentation  , news article...), cite me or this web/post. Yep, the image is real. He is flying from the branch in style

travel

Madagascar: humans

Some pictures taken in Madagascar. Use them as you  please with only one  restriction: if used in public (blog, presentation  , news article...),  cite me or this web/post.

travel

Madagascar: Landscapes

Some pictures taken in Madagascar. Use them as you  please with only one  restriction: if used in public (blog, presentation  , news article...),  cite me or this web/post. Politics

kafka

Using Avro In kafka Streams: the hard way

In this fourth part we will see how to use avro in kafka, and we will try to do it withhout using any extra help, the old way. This way we will experience the pain of not having a good integration ecosystem (e.g. a native library and the schema

kafka

Producing JSON data to a Kafka topic

The first big step to work with Kafka is to put data in a topic, and so is the purpose of this post. Being JSON the most common way to intercomunicate, and having the schema with the data, we will explore how we build a producer in scala to start