Python is a general programming language. Unlike R or MATLAB, it was not designed specifically for data analysis and visualization. This is why most of the code that we will use today comes from code libraries that have been created for these specific uses.

Although many operating systems come with a version of Python pre-installed (including Ubuntu, MacOS, etc.), its configuration, the installation of libraries, etc. can be relatively perilous for a beginner.

This is why for this workshop, we will work with Python on Google Colab (https://colab.research.google.com/). Colab is a service offered by Google, where you can create a Jupyter notebook in which you can mix code (Python and others) and text (in Markdown format). As all the necessary software is already pre-installed, we will save a LOT of configuration time.

We can also, immediately, check that our notebook works well by launching a first test command

That said, if you ever want to work directly on your computers, I advise you to install Python through a library manager (eg Anaconda). This way of doing things will allow you, among other things, to avoid version conflicts between your different projects.

Python can be used as a calculator

The exponents are different from those we often see (i.e. 2^5)

Results can be saved in an object

We won't have time here (obviously!) to explore all the possible data structures and functions of Python.

We will still take the time to get to grips with one, which should be the most useful, namely the lists

The various elements of a list can be accessed from their index, which starts at 0 (rather than 1, it's an old programmer war)

List elements can also be modified

And added

Notice in the previous example the use of the syntax object.function.

We won't go too deep into these details today, but since Python is an object-oriented language, each of the objects has a series of actions (functions) they can perform and associated data. To access these, we use the notation object.method or object.data.

You can also delete items, either by position, or by their value directly

Note that if you use the by-value method, only the first value will be removed.

To automate things in Python, we can, among other things, use functions and loops

We can for example print the name of each species like this

Note that, in Python, whitespace usage is part of the language structure, i.e. it is not purely cosmetic as in other languages.

All text that is indented is part of the loop. No need for braces, parentheses, etc. It's very elegant!

We can also create a function that, for example, checks if our species is a spruce

Another important Python data structure is called data dictionaries. While lists only contain series of values, dictionaries contain pairs of key-value associations.

They are defined using braces rather than square parentheses.

We still access the elements of a dictionary using square parentheses

We can get all the keys or values like this:

It is possible to combine different data structures to form complex objects, such as a list of dictionaries:

Unlike many languages created expressly for data analysis, Python is not a naturally vectorized language.

If we are preparing a list and we want to multiply all its items by 2, we can NOT do this:

We could obviously work around the problem with loops

Python has a more compact notation for this kind of problem : list comprehension

It is a very compact structure, allowing to create a list from another list.

Finally, if you have a lot of vector or matrix algebra to do, it will probably be easier to use the library provided for this purpose, which is called NumPy.

NumPy (and all the other libraries needed for the workshop) are already installed on Colab.

On your personal computers, we install NumPy by the command line (terminal), for example like this:

pip install numpy

In Python, when you activate a library, all the loaded code is associated with a prefix, which can be modified at the time of activation

There is a library that manages data frames much more appropriately than lists or dictionaries when it comes to statistical analysis of data. This library is called pandas.

It takes its name from a common data structure in economics, PANel DAta.

The pandas function for loading data is called read_csv.

To simplify, we will load data directly available on the web

If your data is in Excel format, there is also a read_excel function allowing you to read them directly, without going through the CSV format.

Once the data is loaded, it is important to check the contents our data, for example with the info method:

In ascending order

Or in descending order

Selecting rows

There are several ways to select rows or columns with pandas, but the most general way is to use the loc operator.

The latter can accept either a list of Boolean values, or labels or a sequence between two labels.

This operator also allows you to simultaneously select a list of columns AND rows in the same operation.

Note that, if only one column is accessed at a time and the column name is a valid Python identifier (no spaces, no accents, etc.), it can be accessed directly with a . variable rather than ['variable']

The principle is the same for selecting columns. On the other hand, we must at this time specify which lines we want to obtain (or specify all the lines, like this)

And as discussed, we can combine the two operations in the same command:

Note that the original table is not affected by these operations. If we want to keep the result, we have to keep it in an object

Adding columns

To get an overview of all the descriptive statistics of our data table, we can use the describe function

If we need finer control, we can specify functions and columns manually

We can also apply our summaries to specific groups

The classic module for data visualization with Python is called matplotlib. In general, people especially use the collection of functions named pyplot, which aims to replicate MATLAB's plotting interface in Python

To create a frequency histogram, the function is called hist.

As matplotlib and pandas are extremely well integrated, there is, for most visualizations, a way to call them directly from the data frame

You can then make changes to this graph (or the previous one), then ask to display it again

If you ever work with matplotlib in a script rather than on Jupyter, you will need to call plt.show() to display your plot. Here, Jupyter does it for you.

If our data is qualitative, we use a bar chart instead.

Unlike the previous example, here we must first calculate the counts for each class

To look at the relationship between two quantitative variables, we of course have access to the classic scatter plot

And I'll quickly show you how to change the color of the dots because I had a little trouble figuring out how to do it for a categorical variable.

The key is to first create a dictionary associating each species with a color.

Then when creating the graph, associate each species value with a color value with the map method.

If you want to explore the relationship between a qualitative variable and a quantitative one, there is obviously the boxplot

If you ever miss ggplot2, the entire library has been re-implemented in Python in the plotnine library.

Note that there are still some small nuances vs. the R code. You will not be able to directly copy and paste your code.

Classic statistical tests in Python are provided by the collection of stats functions in the scipy module

If you want for example to apply a T test for unequal variance

If you want to measure a Pearson correlation (and get the corresponding p-value)

Most Machine Learning algorithms are available in Python through the scikit-learn module

Here is how, for example, we could apply a Random Forest classification model to our data.

Note that we must first encode our qualitative variable in numeric so that the model understands it...

We can then run the model and see how well it works!

The sklearn library contains a function to calculate a principal component analysis named PCA in the decomposition module. PCA is there, since it is, in data science jargon, an unsupervised learning technique.

Note that the function always applies its calculation to the variance/covariance matrix. You will have to center-reduce your data yourself to use the correlation matrix.

A function to perform this operation (StandardScaler) can be found in the preprocessing module of sklearn.

Here's how to extract the eigenvectors

The eigenvalues

And the variance explained by each axis

Finally, we can also use matplotlib to visualize the first two axes of our PCA like this:

You can also plot variables in the new axis system. However, you have to use a loop to do this, because the plt.arrow function only draws one arrow at a time

As this is not a statistics workshop, I leave you the pleasure of scaling the two data (eigenvectors and points) in the same graph with the scaling of your choice ;-)

Here are two resources that have been useful to me in building this workshop:

If you really want to learn Python, the most recommended book right now is Python Crash Course, 2nd edition, by Eric Matthes. Beyond the first 200 pages teaching you the basics of Python, the other 300 are devoted to 3 projects showing you the different facets of the language: a game including a visual interface, a data analysis project and a Web application.

If you ever want to apply what you have seen in your statistics courses with Python (and for each piece of code see the equivalent in R), I recommend Practical Statistics for Data Scientists, Second Edition, by Bruce, Bruce and Gedeck. It is both one of the few books presenting examples in two languages and it offers, in my opinion, just the right level of detail for someone who wants to apply good statistical techniques correctly, without having the ambition to become a full-time statistician.