Spark Easy way to accessing s3 buckets with spark It's sometimes difficult to access s3 files in Apache Spark if you don't use a prebuilt environment like Zepelin, Glue Notebooks, HUE, Databriks Notebooks or other alternatives. And googling around might get you half working solutions. But do not worry, I'll show you how easy it is to do it
Spark Loading packages in pyspark jupyter notebook/lab Using spark in an interactive way it's a bit cumbersome sometimes if you don't want to go to the good old terminal and you decide something like a jupyter notebook better suits you. Or you are doing a more complex analysis also. If that is the case and you don't
ai Enable GPU access to WSL2 debian for your AI workloads This post aims to centralize the information on how to use your Nvidia GPUs on debian using WSL2 in order to train and run your AI/ML models.
Spark Grid neighbour operations in Spark In physics or biology you sometimes simulate processes in a 2 dimensional lattice, or discrete space. In those cases you usually compute some local interactions of "cells", and with that, calculate a result. An example of this could be the Ising model which was proposed in 1920 for ferromagnetism in
travel Canary Islands: Fuerteventura Some pictures taken in Fuerteventura, Canary Islands. Use them as you please with only one restriction: if used in public (blog, presentation , news article...), cite me or this web/post. The consequences of uncontrolled planning and an economic crisis.That is NOT sandThis is actually calcified coral. "Palomitas"Why there
privacy Com esborrar casa teva de Google Maps Fa uns quants anys haviem de mirar mapes o fins i tot preguntar a desconeguts (on s'ha vist!) per poder arribar a llocs nous o que no recordàvem on eren, i ara només hem de buscar el nom del lloc i automàticament ens diuen on és i quina és la
travel Canary Islands: Gran Canaria Some pictures taken in Gran Canaria, Canary Islands. Use them as you please with only one restriction: if used in public (blog, presentation , news article...), cite me or this web/post. Dunas de MaspalomasThat is an odd museum, its purpose is to show the "scientific" studies done on native canary
ai An overview on the EU Proposal for an Artificial Intelligence Act Recently the European Commission released their proposal for a regulation laying down harmonized rules on Artificial Intelligence. After a review, I've tried to summarize its contents and explain what is considered an AI system, its classification, measures and finally give my opinion on the proposal. Fist we need to start
travel Canary Islands: La Gomera Some pictures taken in La Gomera, Canary Islands. Use them as you please with only one restriction: if used in public (blog, presentation , news article...), cite me or this web/post. El Teide is always present
travel Canary Islands: Tenerife Some pictures taken in Tenerife, Canary Islands. Use them as you please with only one restriction: if used in public (blog, presentation , news article...), cite me or this web/post. Wildlife
python 3 days deep dive with building websites and API in python and flask For a personal project I've recently spend 3 days learning and coding a website+API with flask. Those have been some of the more enjoyable days I've had in a while (referring to coding). In this post, I explain you my takeaways on how to build better architected apps with
wsl WSL handy commands Often WSL2 does not work well anymore, and those commands have saved me some time and headaches. If you have messed-up something (e.g. sudo), login to the WSL as root (from windows CMD) C:\>wsl -u root Restart WSL2, sometimes needed when you can't connect from your windows to
datascience Classifying architectural elements using AI (ML and neural networks): problem statement I've recently finished the IBM's Avanced data science specialization [https://www.coursera.org/specializations/advanced-data-science-ibm] and as capstone project I thought about being able identify the architectural elements present in an image: is it a column? A vault? Or a bell tower? Let's see what can we achieve! The problem
tensorflow Tensorflow model fails to train with initialize error This post will provide some troubleshooting resources if you find yourself with this UnknownError when training an algorithm: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. Environment: * Windows 10 * Nvidia RTX GPU
Spark Set-up pyspark in Mac OS X and Visual Studio Code After reading this, you will be able to execute python files and jupyter notebooks that execute Apache Spark code in your local environment. This tutorial applies to OS X and Linux systems. We assume you already have knowledge on python and a console environment. 1. Download Apache Spark We will
opinion Featured The Theology of Artificial Intelligence Artificial Intelligence systems are every day more present in our lives. Despite interacting continuously with them we are usually not conscious about how they work or what are the consequences of using this. In our mind, it's just a black box.
Spark Compute 1-d histogram in spark Spark has a way to compute the histogram, however, it is kept under low level and sometimes obscure classes. In this post I'll give you a function that provides you with the desired values passing a dataframe. In the official documentation the only mention [https://spark.apache.org/docs/latest/
Spark Remember to execute the windowing once with Spark This is one of the things it makes sense when you stop to think about it. When you are performing the same aggregation over a window in more than one column, it is recommended to execute only once that windowing. Lets dive in with an example: I am working with
Spark Configure Nginx as reverse proxy for SparkUi Nginx is a common webserver to be used as reverse proxy for things like adding TLS, basic authentication and forwarding the requests to other internal servers on your network. In this case, we are going to serve the sparkUI adding security (https) and authentication, and serving it in a different
Spark Obtaining latest PK value Windows vs Joins with spark When working with change data capture data, containing sometimes just the updated cells for a given PK, it is not easy to efficiently obtain the latest value for the entry (row). Let's dive into an example. First though, we need a couple of definitions about change data capture (CDC): > Change
travel Madagascar: Animals Some pictures taken in Madagascar. Use them as you please with only one restriction: if used in public (blog, presentation , news article...), cite me or this web/post. Yep, the image is real. He is flying from the branch in style
travel Madagascar: humans Some pictures taken in Madagascar. Use them as you please with only one restriction: if used in public (blog, presentation , news article...), cite me or this web/post.
travel Madagascar: Landscapes Some pictures taken in Madagascar. Use them as you please with only one restriction: if used in public (blog, presentation , news article...), cite me or this web/post. Politics
kafka Using Avro In kafka Streams: the hard way In this fourth part we will see how to use avro in kafka, and we will try to do it withhout using any extra help, the old way. This way we will experience the pain of not having a good integration ecosystem (e.g. a native library and the schema
kafka Producing JSON data to a Kafka topic The first big step to work with Kafka is to put data in a topic, and so is the purpose of this post. Being JSON the most common way to intercomunicate, and having the schema with the data, we will explore how we build a producer in scala to start
